Ted Herman ]] Professor
Department of Computer Science
201M MacLean Hall
The University of Iowa
Iowa City, IA 52242
Tel. (319) 335–2833
Fax. (319) 335–3624

ted-herman [at] uiowa.edu

Research Interests

My research interests are: applied sensor network technology, especially clock synchronization; distributed computing, particularly self-stabilization.

My teaching wiki is at http://weblog.cs.uiowa.edu .

Big Data

Here’s a Q&A site for data geeks: http://metaoptimize.com/qa/

Social Networks

Here’s a TED talk about the intersection of epidemiology and social networks: http://www.ted.com/talks/nicholas_christakis_how_social_networks_predict_epidemics.html.

As an introduction to the general areas of social networks and social software, I found the blog by Dr. Michael Wu, one page of which is here: http://lithosphere.lithium.com/t5/Building-Community-the-Platform/Social-Graphs-The-Art-and-the-Insights/ba-p/5713.

Progress on August Flash Decoding

8/27/10 - converted the XLS files that Tina gave me to CSV, wrote a small Python script to read in and build a record for each line of the file; next step is to figure out how to deal with these files — part of the information is in the file name rather than the spreadsheet :(

8/27/10 - rough design, first steps: construct formal maps of the metadata, which is partly in spreadsheets, partly in filenames/directory structure, partly in graphical images … all this needs to be put into tractable form. my plan is to organize things temporally around shift numbers (1,2,3,…) and then have the shift number be a key into a list or dictionary for other information. example: a dictionary for time can map shift number into start, stop (those would be date objects) and offset (relating standard date/time to mote counters of seconds since start of experiment); another example: a dictionary mapping (mote-id,shift) to some name for its location; in turn, the location name should be mapped to more detailed coordinates, room names, etc. we also need a table for the semantics of which puck is the hand cleanser for which room and vice versa (not always a bijection).

8/28/10 - I took the morning off from working on the semantic data (spreadsheets, informal knowledge and other information) to explore the flash data and re-craft the conversion tool. In doing so, I attempted a first pass at a filter to remove bad flash files. The total raw flash data, when compressed, is about 135MB, but triples in size when decompressed. I put all the files into one directory for convenience in batch processing (David - do the filenames contain enough information to identify when/who the flash is from?). The filter only looks at the contents of one mote’s flash and tries to make sense of it. There were these possible errors:

• Empty or very short file; the tool discards the final 240 bytes to be conservative, because TinyOS on the motes could badly truncate the final buffer when spontaneously turned off or the battery dies.
• “Fence” error: the log format has some special characters to recognize the format, and if these aren’t in the right places, the flash is probably corrupted (though, in the case of DoorMinders, it’s due to a different format).
• Timestamps go “backward” — this should never happen in life, time going backward. I speculate this is either failure to clear flash and/or it could be that motes synced up among themselves in some isolated area during reprogramming, before being deployed.
• A large (over one hour) gap between timestamps. This would either be a scheduling error in the software, which is supposed to record timestamps once every 20 minutes, a failure to write a log record, or a clock jump due to synchronizing to a small timestamp, then jumping to the right one as a mote joins the group in deployment. Below, this is labeled as a failure to synchronize.
• Absent time of at least one hour. This could be normal, or could be a troubling sign, I don’t know. I will work on the assumption it is normal, and we should try to use the data (unlike the other cases above).

My plan next is to return to semantic data problems. For future work, the conversions could do some cross-validating of the times using trigger events (hand washing signals), which likely get received by multiple motes and should be timestamped the same. I noticed a small discrepancy between the per-event log records (16-bit clock values) and the timestamp records (32-bit full clock values in seconds), amounting to about a minute or so. I haven’t had time to investigate this yet. The converted format shows a full, 32-bit clock value for each record and omits any timestamp records. Also, in future, Don could tell me what logical times are related to real time, and this could be put into the format of the converted files, as well as skew compensation (motes run slightly faster than real time).

Results are on vinci, under a “converted” directory of wescrub.

8/28/10 - I now have some semantic data for the stationary (non-badge/pager) motes. I concentrated on operational status only for starting this. Conceptually, this means there is a table (Python dictionary) which says, for each mote and each shift, whether that mote was working or not, according to the spreadsheets. I hope I did this correctly.

This semantic data will be used in filtering the flash files, to throw out entries that appear to be coming from bad motes. I’m guessing the next step is to choose a few working pucks and a shift, then explore all the flash files for that shift and see how many trigger events occur. If this number is large enough, then drill down. I don’t know if enough is known yet to do this in a completely automated way (since I don’t know which signal levels are meaningful). A few experiments like this are on the agenda for the remainder of today.

Beyond this, of course, would be adding location to the semantic data.

Let me know if any of this is worth putting up on vinci to share — otherwise I will wait until it is more polished.