Progress report

‘Been working hard on BrewSession for a few days and I’d like to tell my readers (or is it reader?) about it. I got the hop data Greg compiled and the BJCP style guidelines back into the database. It feels like an accomplishment because this data is critical to meaningful brewing software. Next up is to move my spreadsheet of yeast information into the database and re-compile the data I had about extras. Today is a holiday in the US so I will be hard at work on that. Our biggest data hurdle is finding malt information and putting it into a consistent format. Greg is working hard on that. Once all our data is loaded and backed up I believe BrewSession will be around one quarter complete.


It doesn’t have to start out so advanced

Merry Christmas. Here’s a little javascript brewsession present.

I was thinking about the ingredients table Greg designed. While it is good looking and we’ll definitely use it, I would rather see a very simple list to start with and give the option to have an “advanced view”. It occurred to me to let the user choose which columns they want to see. So I restarted work on BrewSession again with this prototype. It’s a work in progress, but a solid start.


Disaster Recovery

BrewSession is not off the ground in a professional capacity, therefore all the code is housed on a machine I have running on my home network. A week and a half ago that machine’s disk controller corrupted my data. Then I screwed up the backup copy of that data in the restore process. The data is almost surely still there, but it’s just beyond my knowledge system administration. I have acquaintances in the data-recovery business, so there is hope for a full restore.

I am upset about losing 4 months worth of hard work because we were planing to have a private alpha test next month. However, now that I have sixteen weeks of ruby programming experience recreating the project will not be difficult. With luck we will be able to roll out our alpha in early spring, 2007.

The lesson here is to test your restores before you need to.


Hop bitterness calculations

Since there has to be a first post some time, I thought I’d start. I need a break from coding, too.

All hop utilization formulas are “best fit” calculations to enpirical data – researchers look at results from their expirements and try to find an equation that best fits the data. This reality produces a few different ways to estimate the final bitterness in your beer. For BrewSession Greg and I decided to offer three bittering calculation methods.

Many brewers regard Tinseth’s method to be well suited when doing full boils using whole or plug hops. There are quite a few parts to it, all of which we will post to the FAQ or User’s Guide or Glossary when we get it up.

Partial-mashers and extract brewers are better served by Rager’s method which is more accurate when doing partial boils with pellet hops loose in the boil. We’ll make this formula available too.

Daniels-Mosher is a somewhat new formula. In fact, I can’t find the formula referenced on the Internet (speak up, Greg). We will be sure to include it with the others.

Because we want to use the right tool for the right job, I devised some programatic logic to select a utilization calculation that best fits the hop as used in individual recipe bitterness calculations. Currently there is no regard for full v. partial boils, only hop form – all the information is there, so the code may change if there is a need. The code first checks to see if the user has a preferred calculation method and uses that if present. Failing that, BrewSession decides which method is better – Tinseth for whole and plug hops and Rager for pellets. If BrewSession can not figure out the hop’s form, it uses Daniels-Mosher. Unsure of where to best use D-M, I threw it as the “unknown” case. If you have ideas, hit the comments or drop me an email.


It’s not finished, but it’s over

Well, it’s over. One week ago tomorrow my thesis was due. Did I make it? No. Was it anything having to do with me? No again. Let me tell you what has been going on:

For those of you who don’t know, I’ve been massively, massively stressed since around June, when I realized that I needed to finish my thesis and graduate in December or I would lose a class. You see, San Jose State has a 7 year limit on classes–once a class is over 7 years old, you can no longer be used towards graduation units. I started SJSU in fall of 1999. You do the math. Anyhow, in order to graduate and keep this from happening, I needed to finish my thesis. For some beautiful insight on my state of mind these past few months, read my last blog. I’ve never been more stressed or more on edge that I was these last six weeks or so. So much so, that for the next several years at least I’ll be able to deal with any and all stressful situations by sighing and saying “Well, it’s bad, but it’s no Fall ’06.” That’s bad.

Now, some background. I started SJSU in Fall 1999 after meeting with my advisor (Dr. Jeff Honda–if you’ve read my previous blog, or if you know me even a little, then you’ve heard some version of my rant about Jeff. In short: he sucks). I felt very lucky–I was interested in forensic entomology, Jeff had just finished up a murder case and was all jazzed about forensic entomology, and he wanted graduate students who were self starters. Excellent! I’m not fond of being told what to do (once again, anyone who knows me, or who has had to deal with me after a particularly bad day knows this. Try it some time…I dare you) so this was going to be a perfect fit. Besides, I was finally going to be studying entomology exclusively! Yay! Jeff told me to take the first semester and get my bearings…take some classes, learn the ropes, that sort of thing, and then he and I would start working on my project. Neat! So I did. That first semester was a bit rough. I didn’t exactly have the best study skills (I’m a graduate student! I know this shit!) and while I did great in my larval taxonomy class, I completely failed microbiology. Failed failed failed! It was ugly. However, it got me in gear and while I was on probation for the next semester, which meant I couldn’t teach which was surprisingly upsetting, I got my ass in gear and figured shit out. Besides, now I can tell my microbiology students that I failed that class, and I know how they can pass. They never believe me, but it’s still nice.

Anyhow, I continue on in my credits and sit down with Jeff to figure out my project. It eventually gets going: a survey of forensically important flies in Santa Clara County. It was a neat concept–no one knew what fly species were common here, and since forensic entomology is gaining acceptance a survey would be a great thing. So I take the next most of the year figuring out how to get this field work going. Field work is unpredictable and hard–especially if you have to choose between going home and watching TV or hanging countless traps full of rotten liver water on steep mountain trails. So I procrastinated a bit, then got motivated, then procrastinated, then ran a variety of pilot studies, then procrastinated, then got my data. This whole time, Jeff vacillated between being annoyed that I wasn’t consulting him more and being annoyed that I was bothering him with stupid (his word, not mine) questions about silly things like project design and trapping methods. So I begin my project proper, which, at the time, involved one year (4 seasons) of weekly trapping in 3 areas, 4 traps in each area. I’d check the traps once a week, preserving any flies that were trapped during the week for later identification. In the beginning I was also going to put out dead chickens and compare the difference between the maggot masses developing on the chickens and the adult flies I caught in the traps, but that idea went out the window when I realized exactly how much time all this was going to take. That, and when I realized that Jeff forgot what my project was minutes after I talked to him. I actually went into the lab one day and had this conversation:

Me: “Well, the chickens are out, but I’m having trouble keeping the animals from eat them”
Jeff: “What?”
Me: “Oh, my project. I put out my liver traps last week, and this week I put out the chickens.”
Jeff: “I thought you were putting liver in the traps.”
Me: “Yes, I am, but we discussed putting out dead chickens to compare…
Jeff: “You should put liver in the traps”
Me: “I am. That’s all done. Just last time you said I should put out some sort of animal, and we decided on chickens…”
Jeff: “Yeah, just put liver in the traps. Don’t worry about putting chicken in them. I don’t think chicken will attract flies anyhow.”
Me: “Um, OK. I’ll do that.”

So I stuck to the 12 liver traps (STANKY!!) that I checked once a week. This process took hours! Hours and hours! I drove from trap to trap, cleaning it out, collecting the flies, putting the flies in alcohol, and putting new liver in the trap. On average, I was doing 5 hours of field work to get this done. It doesn’t sound like much, but let me tell you, when you’re taking a Sunday morning (OK, afternoon) to drive around and play with rotten meat, and you still have to go to work and do hours and hours of homework, it’s a lot. Needless to say, it wasn’t fun. I started pinning the flies, but found that it took even more time. I was collecting several thousand a week, and it could pin about 80 an hour. So I did a few (usually when we were watching a lot of sports or something) but I didn’t have the energy to embark on total curation of all flies yet. Anyhow, the trapping was going along swimmingly for several months; it was fine, that is, until Jeff started talking about extending the project. No! I don’t wanna! This is painful! So, in my oh so mature way, I started avoiding him. I figured that if I could make it through the year without him extending the project, I could terminate it and it would be OK. I made the mistake of going into the lab on a Saturday to get more vials, though. There he was, lying in wait as Dean and I wheeled our tandem into the building. “So” he says “I think you need to extend your project by another year.” Ever get that heart sinking feeling? Yeah, me too. Me: “But the year’s almost over, and since it’s going to take me so long to do the identification…” Jeff: “Nope. Extend it by another year or so. See you Monday.” Damn.

So this goes on for another year. Granted, it did make it a better project, but I’m of the mind that the project didn’t need to be perfect at the expense of years and years of my life. Maybe I’ll get the nobel prize for this thing. Here’s hoping!! I’m not good at arguing with authority figures, however, so I didn’t argue that much. Besides, I did like doing the project. Now I’m 3 years into the research, and the field work is finally, finally over. Next, I tackle the daunting task of pinning all these damn flies. There ended up being 40,404 blowflies, plus a bunch of of other types. I had some help for a while, but the undergraduate Jeff gave me kept losing buckets full of flies (“I’m not sure what happened! I think I left them outside, and today they’re just gone…”) or kept letting them sit so in water so long they would mold. I lost more data that way…and for the time I put into this, I decided it wasn’t worth it. I took on the curation of the flies myself. This entailed soaking the flies in a restorative liquid (the alcohol I had preserved them in causes them to shrink when they are dried out. If you soak the flies for an hour or so in Stoli, yes, the vodka, then they puff back up. Weird in an intoxicating sort of way) then pinning each one. I spent 9 months doing this. Hours and hours of sitting in the museum, listening to books on tape, eating Top Ramen and pinning flies. When my fingers got so pruney the skin started coming off, that’s when I’d go home. It took forever. However, it was enjoyable compared to what was coming next.

After all the pinning was done, I had to identify these little buggers. That’s much harder than it sounds. I’ve taken several entomology classes, I’m familiar with the dichotomous key used to identify insects, but I’ve never been taught the complex language of fly anatomy. I spent many, many months desperately trying to figure out what phrases like “Greater ampulla with stiff setae; dorsum of first and second abdominal tergites black, posterior margins of abdominal tergites 3 and 4 black” meant. And this was just to tell families apart–I still needed to figure out how to tell the different species apart. Why didn’t I consult my advisor, you ask? I did. Twice. Both times, I had very memorable, if very short, conversations with him. The first time, I wanted to know which key I should use to key the flies out to the species level. I had a couple of old ones, but I wasn’t sure if that is what I should be using. I saw him in the hallway and it went something like this:

Me:”Hey Jeff! I’ve found this key from 1940, but I’m not sure if it’s the most up to date, and I don’t know where to go…”
Jeff (in a disconcertingly loud voice):”GODDAMN IT ADRIENNE! THIS IS NOT ROCKET SCIENCE! FIGURE IT OUT!”
Me: “Um…*small sob* Ok.”

That was the end of that. Eventually he found me and said he hoped I wasn’t crying anymore. He also found a key he thought I should use, and all he had to do was search for it on the internet. He just didn’t understand why I couldn’t figure that out for myself. Neither could I, for that matter. I had seldom felt so stupid. You know what’s annoying? When you desperately want to please someone, and that someone is a bastard. I know he’s a bastard. I know it! Yet I still want to make him proud. I hate that.

The second time I tried asking him for help was several months later when I was staring at the key and trying to define some of the characters I was supposed to see. I knocked on his office door and:

Me: “Hey, what’s a transverse suture?”
Jeff:”God, you’re stupid. Figure it out.” (Slams door).
Me: blink

So I stopped asking him for help. Eventually I guessed that he really didn’t know what a transverse suture was, but that didn’t help me then. My saving grace came in the form of Dr. J. Gordon Edwards (the “J” doesn’t stand for anything…his parents just liked how it looked). He was the old retired entomologist who still ran the museum. He came in nights and took care of things. I could leave him notes with questions, like “What’s a transverse suture?” and the next morning I’d not only have a written answer (Transverse suture: the demarcation between the prothorax and hind thorax, often seen as a horizontal line or suture) but a photocopied packet of drawings with the area highlighted, a book or two explaining the origin and naming of the part, and several reference specimens set out with little notes. This was a fabulous man. He retired at 80 after teaching at SJSU for a million years. He died at 84 while rock climbing in Glacier National Park; he had a heart attack and died on the side of the mountain. I consider him my entomology mentor.

So, with Doc’s help, I slowly but surely figured out how to identify my flies with some sort of certainty. It took me over a year, but I got them all done. Then loomed the task of figuring out what this all meant, and that meant putting all the flies into a database in a variety of ways, and analyzing these data using the magic of statistics. At this point, Jeff tells me that I need two more professors on my committee–I have to have a committee of 3. Um…what? Dang it! I ask who he suggests, and he says I need one stats advisor, then someone else. I talk to Dr. Shannon Bros, the go-to person for stats in the department, and she says she’ll be happy to be on my committee! One down, one to go! I have no idea who else I can have read my thesis, so Jeff talks to some of the other professors and suggests Dr. Mike Sneary. Great! One task down. I talk to Dr. Sneary, and he wants to take a look at my thesis proposal to see if he approves. Wait, what’s a thesis proposal? I ask Jeff, and he says, “you know, the thing you write up to propose your thesis.” I say “I didn’t do one of those….” Jeff says “Oh.” The end. I eventually ask him what it entails and he says “Oh, just write up a proposal.” By this time, I’ve been yelled at enough to not want to try and clarify things he’s not giving up willingly, so I drop it. I spend the rest of the summer wading through books in the library trying to write up a thesis proposal. Granted, once the proposal was written, it did help me organize my thesis. It actually would have been smart to write one before I even started the project, but I had no idea. Live and learn.

I gave my proposal to Dr. Sneary who had lots of suggestions about project design and whatnot, so I had to tell him I had already finished the project (he hadn’t realized that) and I was just starting the writing phase. He told me to send him a polished copy of my thesis once Jeff signed off on it. Ok, check. That takes care of that. I then start my materials and methods. I spend many, many, many, many hours researching and writing and rewriting, and having people read, and rewriting this section. You see, one of my friends in the lab had just graduated, and he said that if you didn’t give Jeff a very polished version of your writing, he got awfully mad and wouldn’t help you fix it. He just made you go back and redo everything. So, wanting to avoid that horror, I worked very, very hard and made this section of my thesis the best I could. Good news: when I finally got up the guts to give it to Jeff I got this note: “Adrienne: looks VERY good. See me about some changes. Jeff.” AAAAAAHHHHHHH! That was one of the best feelings ever. Yay! I was literally walking on air for weeks after that note, and I didn’t want the feeling to end, so I put off any more work on my writing until after Christmas. I was gonna enjoy the praise, damnit!

During this time, I also started working on my stats. of course, right before my first meeting with my stats advisor Dr. Bros, Jeff tells me I should have a different professor do my stats. I say OK, contact the other professor, who says she’ll be happy to help me, and cancel my appointment with Dr. Bros. Jeff then changes his mind (or forgets…I haven’t decided yet) and tells me to go with Dr. Bros. Whatever! Just get me done! So I go and see Dr. Bros during her office hours.

Let me tell you about Dr. Shannon Bros: she’s a transsexual woman…she used to be Dr. Bill Bros until she had the snip and took the hormones. An interesting thing about these hormones–they make you go through puberty again, with all the angst, and confusion, and flakiness that plagues every puberty anyone has ever experienced. So Dr. Bros tends to be a bit flighty. Really, really nice and helpful (when you can catch her) but flakey and an awful lot like a teenager. That’s annoying. She also has a million students and one office hour a week. So I go to her office hour and wait in a line of 9 other students. When she finally gets to me, she only has a minute to talk. I give her a brief overview of my thesis, and she says I need to do some magical stats on it, and have I ever taken the class? No, I say. Should I? No, she assures me. She’ll walk me through everything, but since she has to go and teach she’ll give me some labs to read through and then I should call and make an appointment. She hands me a print out of something called ANOVA and then rushes off. I read the printout in total bewilderment (I haven’t taken any stats since my first year in undergraduate school) and then I wander to the library to try and figure things out.

This pattern goes on for many, many months. I read some confusing lab about some complex statistical test, I make an appointment with Dr. Bros who goes over it with me, then she gives more print outs to read. Eventually she starts sending me to the computer lab to play with fake data and learn to use a variety of computer programs. A semester and a half passes, and I still haven’t even STARTED looking at my own data yet. And I still don’t really understand how to set anything up–the labs she keeps sending me are all using data that is already set up in the program. I finally tell her I want to set my own data up. She says great, and she’ll email me a document that walks me through setting things up. She does this while I’m sitting there then sends me on my way. I immediately run to my office to print the thing out, and find she forgot to attach the document. No problem, right? I email her and ask her about it. No response. I call her. No response. I call again. No answer. I go to her office. Notice on the door: Dr. Bros will be out of town for the next 3 weeks. Please leave a message with the front office. DAMNIT!!

I wait. And wait. Eventually she comes back. She doesn’t have time in her schedule to see me, though, so I have to make an appointment for a few weeks later. I see her. She doesn’t remember my project, what we had talked about, or anything about any document she was supposed to send me. I explain my project again. She says I need to run a test called a PCA. Then she gives me another lab. What?!? What happened to ANOVA and MANOVA?!? She launches into some gibberish about the test and how it will be perfect for me and if I just run this lab with this fake data I’ll understand the meaning of the universe. I nod dumbly and leave in a daze. I spend the next two months searching out all the info I can on ANOVA/MANOVA and trying to set up my data. I finally just do my best, run some stats (which don’t seem to work at all) and email the results to her, asking if they are correct. Three weeks later (while I’m at a conference in Seattle) she sends me an email. No, the stats weren’t right, but she looked over my raw data and since I was on the right track she just fixed the problems and ran the stats for me. Holy CRAP! AWSOME! Now I have this printout of stats that Dr. Bros ran for me, with a short explanation about what they mean. I almost rented a safe deposit box just to house them, but I settled instead on printing several thousand copies and keeping the email in a hundred different places, just in case. So I begin to write my results. They are long and boring and confusing, and I really still don’t understand what is going on with the stats, so it’s very hard to write about. Can you guess what happens? Lots of standing in line waiting for a minute of time with Dr. Bros, lots of canceled appointments because she forgot she was washing her hair or something, and lots of late nights trying to figure out what these stupid math-type words mean.

This brings us to June, when I get a notice from the department that one of my classes is expiring in Fall; I have to graduate or I lose it. I take up the writing speed, once again spending most of the summer in the library trying to get things done. I exchange many, many emails with Dr. Bros, and eventually feel like I have a handle on what is going on. By the time my 30th birthday rolls around, I have a fully finished first draft of my thesis. That, my friends, is a great feeling. I send the draft out to family and friends to proof, and Dean and I go off to my cabin to celebrate for a week. When we get back, I start the endless process of revision. Jeff calls and says he wants to see a version of the thesis by mid August. I comply; he sends it back with suggestions. Well, suggestions may be too strong a word: vague notions might be better. He asks for another draft by September 8th. This is where my last blog started…and where my stress level really began to rise to unheard of levels.

Jumping ahead to this last week. I had been going back and forth with Jeff about a variety of things, the least of which was he wanted a different person on my committee–I’m not sure if he forgot about Dr. Sneary or just started to hate him, but the end result was Dr. Sneary was off and some new guy was on. Whatever. By this time I had neither the energy nor the desire to argue about anything. I just wanted a signature. I give Jeff my thesis, he gives it back, I change things and give it to him again. I even spend the better part of a Saturday afternoon at his house going over my results section, which he had called “unreadable” and “impossibly horrible.” At the end of this meeting, he tells me I need to go to Dr. Bros who should sign off on the results section before he will.

I send my thesis to Dr. Bros that night (two weeks until my thesis is due, by the way) and ask her to read the results section, and remind her that my deadline is fast approaching. I hear nothing, NOTHING, for a full week, despite my many, many emails, phone calls, psychic messages, and attempts to corner her. Finally she sends me an email saying that there are some significant changes that need to be made and I should come and see her in her office. I get this email on the Tuesday before my deadline. Her next office hour is Thursday at 8 am. I get to her office at 7:30, and join the line of 4 people that has already formed. What did these people do…camp here? Hmmmm…I should totally try that. So, in the last two minutes of her office hours I stick my head in, and she tells me to come back that evening at 5 to go over everything. I remind her that my deadline is Monday. She says “you should get it done by then. Even if it’s not perfect, get it done by then.” I leave, and wait in my office ALL DAY because I want to make sure she doesn’t leave the building without talking to me. At 4:30 I set up camp outside her office and wait for her arrival. She shows up 15 minutes late, but dutifully sits down and goes over the sections with me. Yay! Actual progress! And we make really good revisions. She gives me really good advice. She tells me how to fix some figures that were giving me problems. I’m so very, very happy! The end is near, I can just taste it! Then she picks up her phone, dials a number and leaves a message: “Hi, you’ve reached Dr. Shannon Bros. I’ll be out of town until November 17th. If you need to reach me, please leave a message.”

Wait. My deadline is on the 13th. Dr. Bros just gave me a bunch of revisions to make before she’ll sign my thesis, and she’s going out of town. OUT OF TOWN?!? I quell the quiet panic in my stomach (have I mentioned the intensity of the migraine/stomach ache combo that I’ve been sporting for the better part of six weeks at this point? No? It sucked) and say nonchalantly “So, when you leaving?” She replies “My plane leaves at 5:30 tomorrow morning.” Ah. It’s 7 pm now, at least 12 hours worth of revisions to make, and her plane leaves in 10 hours. Well then.

She blathers on for awhile about her trip, I timidly ask about getting my thesis signed, and she says that once it’s all ready she needs to sign the first page. With that, she flits out of her office and asks me to lock up when I leave. I sit there for a moment, mildly stunned, and slowly realize that one of my advisors went out of town and there is no way, no matter how hard I try, no matter how much sleep I lose, no matter how much I pray, there is absolutely no way I will get my thesis signed off in time.

But it’s not my fault (and eventually I..ll believe this). I did everything I possibly could to get this thing done. I worked my ass off. I put off seeing friends and family, put of doing things I like, doing things I needed to do, stopped playing my harp, stopped playing video games, stopped reading books. I did everything physically possible to get this done, and it all came down to my advisor leaving town. At that moment, my migraine went away. I was suddenly not at all nauseous. I walked home happy for the first time that semester. I stopped at a computer on the way home and found that I could get my advisor to sign a paper that renewed the class I was gonna lose, so that wouldn’t be a problem.

So, the point of this long, long, long story is that it’s over. I am no longer trapped behind a computer writing for my life. I am no longer throwing up breakfast or downing shots of vodka to go to sleep at night. I get to go out again (and I have–I’ve been spending the weekends drunk off my ass lately. It’s like I’m in college again!!) and I get to enjoy Christmas. Thanks for reading, and email me! I can email back now!

It’s not you, it’s me

My thesis is due. In 20 days. We’re not talking due to my advisor for sign off…no. It’s due to the department, ready for binding, with all three of my committee member signatures on it, all spelling errors fixed, and printed on super expensive paper. 20 days from today. I gave my advisor a copy of my thesis on Sept. 8. Last Monday he said he hadn’t read it yet and wanted a revised copy by last Friday. Done. Nine 0’clock on Sunday he emails me some revisions which say things like “I hate this section. Rewrite it” or “I don’t agree with your conclusions. See me and rewrite your discussion. Make these changes by tomorrow.” So I tried. I tried really really hard to make the changes. But my desktop computer, you know, the one that I’ve been using for the past years to store my data, write my thesis drafts and print everything, has decided that it doesn’t want to allow me to use Word, or any other text editor for that matter. So Dean (whose birthday it was by the way, and he was a tit bit tipsy) gave me his windows laptop–which decided it didn’t want to open anything from my version (or my advisor’s) version of word correctly, so all the formatting was off, none of the 20 figures were present, and nothing was italicized. So I tried using my Mac—which decided that Dean’s lap top had the right idea, but wanted to one-up it by not having any of the correct fonts installed and making sure I couldn’t change the margins (which were 3 inches). So I tried on Dean’s desktop. He has a linux box, or something of that nature. It didn’t even TRY to open my document…it just gave me a blank page. So I tried Ben’s computer. It apparently had talked with Dean’s desktop and they decided to show a united front. Now it’s been 2 hours, I work in the morning, and I still haven’t been able to make any changes to my thesis. You know that feeling when there’s a ginormous tumor growing in your chest that then reaches into your brain stem and uses your hypothalamus as a punching bag? Yeah, I had that felling.

Of course, if you don’t know my advisor, then you won’t understand. Let me tell you about him. Dr. Jeff Honda is the only entomologist at San Jose State University. He had just be hired when I interviewed for graduate school. Everyone there said that he was great and they all loved him. When I actually entered graduate school a year later, however, all their tunes had changed. Apparently, that “good teacher, nice guy, great mentor” thing he had going on during his interviews was all a front and he’s really just a great big bastard. I’ve learned over the past three quarters of a decade that Jeff takes offense very easily, holds a grudge for years on end, is willing to ruin people’s careers for retarded reasons, and can’t handle any type of opposition without blowing up. Oh, and if he’s in a bad mood for any reason what-so-ever, he’s perfectly willing to take it out on whomever is nearest (and he’s often in a bad mood). About half way through my master’s program, I thought about changing advisors. Jeff has this habit of not helping with any project or advising on any school stuff, or really knowing what is going on with his students at all. He says it’s because he wants self starters and people who can work independently. He doesn’t–he wants students who magically know what to do and then can read his mind and do it his way. Anyhow, I was tired of just making shit up then being yelled at when I didn’t make the right shit up. So I emailed some entomologists at universities and asked what they looked for in a PhD student. The one thing in common? They really liked seeing a great recommendation from their master’s advisor, and that recommendation carried even more weight if that master’s advisor was an entomologist. I can understand–as an advisor you’re taking on an unknown student and promising to mentor and teach her for some ungodly amount of time. You want to have as much guarantee that the student will work hard and do what is necessary to make it worth your while. But what I also took from this was that I’d have a much better chance of getting into an entomology program if my master’s advisor was an entomologist. Jeff is the only one at SJSU. So, I stayed with him (despite all the advise to the contrary from friends and family). I tried to look at it as a lesson in how to deal with really difficult people. I know lots of scientists, and some of them are big old jerk offs. I’m gonna have to deal with them eventually, so I may as well learn how now, right? So I was still Jeff’s student, just struggling with a badly designed experiment and sketchy support at best. Then comes along another student-let’s call her Laurie. Laurie didn’t know how to deal with big egoed men who weren’t good at interpersonal relationships. Laurie got into many, many fights with Jeff over her thesis work. After every fight, Jeff would hold a grudge for a long, long time, and not give Laurie any advice, or sign any paperwork Laurie needed signed. Laurie went to the Dean of biology to register a complaint. A warranted complaint. One that was long overdue. Jeff got called into a meeting. Jeff left said meeting very, very angry. None of us got anything signed off for an entire semester because of this meeting. None of us even talked to him about anything but how horrible Laurie was for the entire semester. I ended up taking art and aerobics that semester because I had no idea what else I was supposed to do, and I needed my advisor’s signature for any of the higher level science classes. Laurie decided she was going to change advisors. She did. Good for Laurie! However, since she did her research under Jeff’s supervision in Jeff’s lab, she needed Jeff’s permission to use any of the data she recorded–it’s technically Jeff’s data, and any paper or thesis that results from the data has to have Jeff’s name on it. Can you guess what Jeff did? Yep, he refused permission. Laurie had been working on describing a new species of microscopic wasp that Jeff collected on one of his trips to Japan, and Jeff withdrew any permission she had to use any of the drawings, pictures or specimens she had been working on for the past 3 1/2 years. Laurie tried for a year and a half to fight him (with many angry phone calls and emails–after each one, someone still in the lab would get screwed over somehow. I got it when I submitted a poster I was presenting at a conference for Jeff to approve, and Jeff didn’t notice that the authorship took up 2 lines…so after my long full name and title, his name was on the second line. He freaked out, I got a 3 page email in all caps about how horrible I was and how I was trying to cheat him out of his rightful publication, I received a formal letter from the department reprimanding me for leaving off his name, I got called into the Dean’s office to discuss the matter, and Jeff took the keys to the museum, where all my data was stored, away from me. This was because he misread the poster. That took forever to get cleared up). Anyhow, Jeff never signed off of Laurie’s project, and after 4 years of work, she had to start anew. She hasn’t graduated yet; I think it’s because Jeff keeps threatening to show up at her defense. God, what a jerk. So that is what happens when you piss this man off. Now you know.

Back to my story. Jeff didn’t read my thesis when I gave it to him. I said nothing. I’ve actually learned a rather sneaky technique–since I know so damn many professors in the department (I’ve taught for all of them), when I see them in the elevator, I talk about my thesis, I mention how busy Jeff is and how I don’t want to burden him with anything else but I just gave him my thesis and I hope he gets around to reading it, then I mention that he’s a great mentor and I’m lucky to be working with him. Then I look stressed and tired and I leave (that last bit takes very little acting…I am stressed and tired). If you don’t know, professors are the biggest gossips. So when I say this stuff, even to random Dr. Bob in botany, it’ll get back to Jeff via the grapevine. It’s not as direct as talking to him myself (which is pretty much guaranteed to make him never, ever read my thesis) or going to the Dean of the department (which is guaranteed to make it so I will never graduate), but it works. This is how I got him to read find me in the hallway and ask for a thesis revision last week. I, of course, mentioned how my thesis has changed since I sent it to him and I’m glad he didn’t read the first because I hadn’t finished all the stats. This got him to promise to read this version over the weekend. Ass kissing takes forever.

So this is why I was so damn stressed about not being able to make the revisions on Sunday night–it’s taken me months on end to get him to the point where he’s willing to read my thesis at all, let alone give me revisions to do with some sort of guidance. If I don’t follow his directions, he will get all pissy and not work with me for a few weeks as punishment. He’ll be “too busy.” Yeah, too busy trying to make me cry. He’s done it before. So I freak out. And then I can’t sleep, because I’m freaking out. Horrible. I wake up after almost 3 hours sleep just before my alarm goes off yesterday and try and figure out how the hell I’m gonna keep him on my good side.

He came into my office hours yesterday (I work for 12 hours on Mondays…I use my office hours to eat breakfast. Not the best time to get my stress level rising). I printed out his revisions from my office computer, and made some vague comment about making the changes on my laptop. Luckily, he believed me. He then when over all the changes he wanted made. Good! At least I know a bit more about it now. There were some major and minor things–and several sentences he just read incorrectly, then tried to revise based on his incorrect readings, then got more confused, so he just underlined them and said “this is very confusing! Rewrite!” He reiterated this sentiment during our meeting yesterday. I nodded and promised to fix them. I’ll fix them by just taking away the revisions. Done! At the end of the meeting, he asks for the updated version by Friday. I counter with “what about Wednesday?” He says “Well, if you want to do it that way…” I nod eagerly. You see, once Jeff decides that the thesis is good enough, I still have to send it to 2 other committee members who will hate it and make me make changes. All this in 20 days.

So, what I’m trying to say with this long-winded tirade, is that I have 20 days to play politics and get some damn good writing done before I lose my graduation standing. I’m not sure if I’m gonna make it, but I’m gonna work my ass of and try. So please, don’t be offended if you are one of the many, many people who have sent me emails that I haven’t responded to, or the hundreds of friends who just want to say “hi” which I ignore. It’s not you, it’s me. I’ll be much better in just under a month, I swear. Come over for International Day of Baking Goodness, November 22. I still like you, I do! It’s just I may murder someone very soon, and I don’t want it to be you. See you in 20 days,



No, I’m not going to effen relax!

For those of you who don’t know, we’re trying to get pregnant. (See EVERY other blog I’ve written for more details). Now, after reading all that background blogging, you should know we’ve had a bit of a hard and painful time accomplishing this goal. I’m not complaining…well, I am complaining, loudly and at length, it’s just a saying…and for the most part people have been truly supportive. But, from the bottom of my heart, and for the sake of every woman in my situation, I have one request: PLEASE, DON’T TELL ME TO EFFEN RELAX!

Ok, here’s where I’m coming from. When you start infertility treatments, it’s not something you do on a whim. Oh no. They won’t even *look* at you if you haven’t been actively trying to get pregnant for a year (and by active I mean charting your cycles and using ovulation tests and all that). If you haven’t, they tell you you, you guessed, start charting your cycles and use ovulation tests. Then they do some basic tests–sperm count, hormone levels, mucus viscosity, that sort of thing. It’s not until several cycles later that they actually begin the pills and ultrasounds. What I’m saying is this: it’s not like you can walk into a doctor, say “I think I’m infertile” and they’ll send you home with a packet of needles and book on test tube babies. No, it’s a long and involved process that weeds out anyone who just hasn’t gotten lucky (well, they’re assuming you’ve gotten lucky a little bit–otherwise you’d need a whole different lecture).

Now, during this year or so of unsuccessful, at home fertilization, you have a lot of sex. I mean, A LOT. As anyone who’s trying for a baby–they’ll tell you it’s been mere hours since the last attempt. They’ll also probably tell you the position they tried and how long she stayed with her legs in the air to ensure proper spermatozoon flow…parents-to-be are a strange breed. Anyhow, lots and lots of sex. During all this sex, you are most likely going to have an orgasm. Well, I’m assuming. If not, sad for you! Let’s just say from experience, you’re going to have an orgasm. Maybe not all the time, but definitely more often than if you weren’t doin’ it several times a day. Do you know what orgasms do? They release endorphins in your brain. You know what endorphins do? They relax you. Completely and totally. Here’s a quick science lesson about endorphins:

Endorphins are released by the pituitary gland and hypothalamus of the vertebral brain. They act on receptors in the brain that, when bound up, cause the body not to feel pain and bring about a general sense of well being. These same receptors respond to opiates and morphine. However, anything injected into the blood stream has a very small chance of getting to the brain because of the blood-brain barrier (the brain is very selective as to what it lets in, so there’s a barrier between blood and the brain. Only certain substances make it in, and only a tiny amount of those substances at that). Endorphins are produced in the brain, so a little bit goes a long, long way. Acupuncture for pain management induces the brain to produce excess endorphins, thereby eliminating any pain in the body. Small amounts of endorphins cause muscles to relax, blood pressure to drop, heart rate to slow, adrenaline levels to lower, and breathing to become deep and regular. They basically put you into an extremely relaxed state–hence the desire to fall asleep after sex. Well, after good sex. Sometimes sex elicits the desire to sneak out without waking the partner and throw up in an alley on the way home, but I digress.

What I’m saying is simply if you have a bunch of orgasms, you’re going to be extremely relaxed most of the time. Even if you get into a stressful situation, residual endorphins will make it seem much less stressful than normal, and allow you to relax much more quickly once the situation has passed. What does this have to do with my bitch-fest above? Read on and all will become clear.

Once you’ve gone through this year or so of bliss and have decided to embark upon the rocky road of infertility treatments (rather than just buy the damn motorcycles and start doing illicit drugs already–a decision many women going through these treatments regret every few months…) then you will begin to notice a strange phenomenon: everyone, and I mean everyone, seems to know how to get you pregnant. You’ll be talking with a friend, say, and telling her about your latest adventure in the doctor’s office. This friend has been on birth control for 12 years, hasn’t had sex for 6 of those, and has no desire to become a mother. Suddenly, she is an expert in fertility, and is happy to share her knowledge with you. Well, fine. I realize that many people show their support by trying to help you out. I actually appreciate it. I listen, take mental notes, look up info I don’t quite understand, and print out resources to add to my notebook. However, after talking with EVERYBODY about this subject, I have noticed one thing in common: they all end their lectures with “You just need to relax. You’ll never get pregnant by being all tense like this. Once you calm down and relax, everything will work out fine.”

This is the point where every infertile woman has her head explode. JUST RELAX?!? Are you kidding me?!? I just spent over a year of my life awash in sperm and endorphins–more relaxed than a pot head on a semester-long bender, barely able to stand upright because my muscles simply refused to tense up, and dealing with situations that make most people bleed out their eyes in frustration by saying “Really? Bummer. I’m gonna go take a nap. Wanna get a slurpee?” and you’re telling me all I need to do is relax?!? Did it every occur to you that maybe, just maybe, there is actually something wrong? That the doctor didn’t cut me open and poke at my uterus for shits and giggles, but because he was taking something out? That I don’t have Dean inject me with painful chemicals because I like that burning sensation, but because I don’t produce them on my own? Did it? I mean, lord, do you honestly think that I’d be putting myself through hell via daily doctors visits if all I really needed was a weekly massage?!? Dear god, shut up! I do not need to relax! We tried that, it didn’t work. Now we have to try something much less fun and much more invasive. I’d appreciate it if you would understand that then either 1) nod sympathetically, give me a hug and buy me presents to make me feel better, then look away when I burst into tears, or 2) find an article on alternative treatments, cutting edge surgery techniques, specialists in the area, or witch doctors who may help, cut it out and hand it to me. Either way, you’re giving me what I need–actual support and friendship during a very, very rough patch in my otherwise extremely happy life. By telling me to relax, you’re basically saying “Any tool can get pregnant…I can’t believe you haven’t figured that out yet! Well, since no one else has let you in on the secret, here you go.” I feel broken enough already! I don’t need you telling me that the answer to my problem is so simple an idiot could do it. You know how that makes me feel? Destroyed. Every time someone gives me this simple, one word answer to my extremely complex situation, it reminds me that I can’t do the one thing my body was designed to do. I can not pull off the single act every other organism on the plant can accomplish. I’m broken. Completely and utterly broken.

So, on behalf of infertile women everywhere–please don’t tell us to relax. We’ve tried that. We’ve tried that for years and it hasn’t worked. Yes, going through these treatments is stressful. Yes, hormones released in stressful situations can inhibit pregnancy. Why do you think the orgasm evolved?!? To relax us when it counts the most! But the one thing that causes my blood pressure to skyrocket more than anything else I’ve been through these past few years is someone telling me “Just relax. Once you stop trying, it’ll happen on its own.” Fuck you. Fuck you all.

Stealing A Buffer

Jumpy, jittery, convoluted video and audio

For a long time, Myth did well. It did everything we expected it to and even a little more. Then shows began to degrade. It may have been a driver or software upgrade or a settings change – these things don’t “just happen” – but I could not trace the cause. Random shows would end up with jumpy audio and video skips. I found a similar description on the web: It is like the first 200ms of each second is missing. Try watching that. Ben came up with a three-level rating system for the severity of unwatchability. Level One was annoying, but watchable and often went away. Level Two neared pure noise and at Level Three you lost all hope.

I turned debugging on, collecting volumes of log files. At one point, I filled up my /var filesystem (how did I forget to turn cron on? Oh yeah, on most other operating systems it’s running by default!) which does not make Myth happy when it’s trying to write logs there.

I tweaked settings, trolled for clues in the detritus, searched for and requested help from the Internet. No improvement. No message in the data. No suggestion availed me.

Recordings are big files

1.3G Apr 4 00:00 1047_20060403230000.mpg
1.3G Apr 5 00:00 1047_20060404230000.mpg
1.3G Apr 6 00:00 1047_20060405230000.mpg
627M Mar 30 23:30 1063_20060330230000.mpg
1.3G Mar 23 02:00 1048_20060323010000.mpg

Behind the scenes the operating system breaks up each file into manageable blocks and writes the chunks to the disk. It tries very hard to put all the blocks in a row so it can read the whole file much like you are reading this text – word by word. When the OS deletes a file it makes the file’s blocks available for other files that may come along. This creates a “space” between two files on disk. Often, new files are bigger than the space created so the OS fills the space with part of the file and then uses new space for the remainder. This is called file fragementation. Fragmentation is bad because it slows down file reads as the OS has to look all over the disk to get each piece of a file. UFS, a widely-used filesystem for unix machines, somehow keeps fragementation from becoming a problem. However, I was using XFS to store recordings.

That made me wonder

I wondered if XFS was as good as keeping fragments out of the filesystem and if fragmentation could cause my A/V problems. I fired up the xfs_db utility to check the situation.

mythical # xfs_db -f /dev/mapper/myth_vg-myth_lv
xfs_db> frag
actual 14468, ideal 402, fragmentation factor 97.22%

This seemed bad to me. The XFS defrag’er only works for two hours at a time, so I set up a little script in screen to do the defrag’ing while I slept.

mythical # while true
while> do
while> clear
while> date
while> xfs_fsr
while> echo done!
while> sleep 300
while> done

When the work was done several hours later, new recordings improved in quality. I was happy with myself and posted the fix to the mailing list.

I hoped recordings would return to normal

I Don’t Watch Commercials

I do not miss the advertisements

Previously, I blogged about how I got to be a MythTV user. Possibly the best thing about Myth is automatic commercial skipping! It is not perfect, but when it works, you barely know your’re watching broadcast television. It works by marking suspected commercials and jumping over those marks during playback. There are some very clever developers that wrote the code to go back over a recording looking for station logos, special video frames, closed-captioning breaks, among other things. This operation takes (on my machines) about 20 minutes per hour of real video and can (currently) happen only after a show is finished recorded. Each flagging job queues up so depending on how many recordings just completed it could be hours before the show is really ready to watch. You can always watch a show immediately after it begins recording. I have become so accustomed to time-shifting all my television that the wait between live broadcast and prepared recording is unnoticable. Watching a well-flagged television show is quite cool and even if the flagger misses, it is an easy thing to rewind then skip ahead in 30 second jumps. We do miss the TiVo sounds just a little, so someone makes them when necessary; ba-doop ba-doop ba-doop.

In addition to the volume of bad adverts, we miss out on the occasional not-so-bad commercial. The benefits and detriments of a comsumer culture is left as an exercise for the reader, but it feels nice to be free of stupid, stupid commercials. Adrienne felt that she was missing out on too much new TV, so she went out and subscribed to a television magazine. I guess people are willing to pay for targeted advertisements.

Yet another MythTV Blog

TiVo was cool, but I wanted something more.

Enter Robert X Cringely, a technology columnist for PBS . He has 20-some years in the industry and writes compelling articles. In late 2004, I read this article which launched me into a homebrew PVR project. Cringely wrote about Andrew Greig, a man in Canada who works for Starnix, an international Open Source software and services consultancy. In addition to providing free VIOP and Wi-Fi to his neighbors Andrew runs a (very large) installation of MythTv. I wish I could see this guy’s setup. Quoting the article:

Andrew’s server runs Myth TV, an Open Source digital video recorder application, storing on disk in MPEG-4 format more than 30,000 TV episodes, movies and MP3 music files. “As each new user comes online, I add another TV card to the system so they can watch live TV,” says Andrew, “but since there are only so many episodes of SpongeBob SquarePants, nearly everything that isn’t news or sports is typically served from disk with full ability to jump forward or back at will. We’ve reached the point now where the PVR has so much in storage already that it is set to simply record anything that isn’t already on disk.”

Think about it. These folks up in Canada can not only watch everything we can watch on TV, on a whim they can watch every episode of the original Star Trek in the order they were broadcast ALL ON ONE WEEKEND.

Doing a little math on a typical 60-minute TV show (at 1.5-2 megabits-per-second) we get 673-900MB per show. If half of those 30,000 files are television shows, I estimate 10-15 TB of storage. That’s a lot of juice to run out of your house. I am not sure how many users his amplified Wi-Fi reaches, but “most” of my suburban neighborhood is probably 50 – 100 homes. And I thought I had IO issues.

So I Set Out

I am a computer dork, so I had some extra hardware around with which to start my project. Taking the best of the leftovers, I put together an AMD 1GHz processor, 512 MB of memory and a 40 Gb disk. Still, I needed a way to get the television into the machine. At the time, I was living with my wife and 6 other roommates in an old Stanford party house. Talking with the roommates about the possibility of watching television from their computers, one of them offered up a capture card to get me started. Coupling the cap-card with a CreativeLabs sound card, I was ready to go.

MythTV is notorious for being difficult

I work with Solaris and live with FreeBSD, so I had a decent idea about the complexity of installing and configuring a unix-like OS such as Linux. I did a quick scan and chose Debian as my distro, primarily because MythTV is developed on that version. My first-ever linux install went fine, as did the MythTV install. Debian’s package management is almost as cool as FreeBSD’s ports. I did struggle for a few days compiling a kernel that had the correct modules and drivers for the sound and cap cards and the rest of my hardware.

The MythBox began recording

So now I could record one channel and store the show in rtjpeg format. While recording the server had encode the stream in software, loading my machine to about 70% utilization. Acceptable, but I could not grow my setup without more horsepower.

What makes MythTV so great

To me, a big draw towards MythTV is its server/client architecture. I ran a master back-end which did nothing but record programs. Because of MythTv I reinstalled my desktop with GenToo linux and the MythTV software. (I also switched the backend to GenToo.) On my desktop, I ran a slave backend (server) and the frontend (client) software. With the extra processing power, my desktop could flag commercials – a CPU intensive procedure – while the backend with the cap card could record and encode television. Any time I wanted to, I could pull up mythfrontend and watch recorded shows (and live tv if the recording schedule permitted).

My wife is a TV Junkie

She was drooling just as much as I was over my new project giving me a high WAF. Adrienne liked the idea for slightly different reasons, but the encouragment was strong. She had me dual-boot her desktop so she could watch recorded TV and surf. With her gentle pushing and the apathetic interest of the other suitemates, I got Samba and dsmyth working so Windows users could watch the recordings in wmp. This got me a second capture card on loan from another roommate. My desktop would now have to pull double-duty as a workhorse and encoding machine. Around this time, I added a 160GB drive for storage to the system.

Launch the Shuttle

For $500, I put together a Shuttle bare-bones system with a DVD RW. Once I got all the right software and drivers on this machine, we set it next to the telly for MythTV bliss. Add a streamzap remote for extra coolness. Now I also had an extra backend system for commercial flagging.

I though 6 tuners would be enough

Towards the goal of recording all chanels all the time, I set out to increase the tuner capacity of my system. Right about that time Hauppauge released the PVR-500, a dual tuner capture card that does MPEG encoding in hardware. This removes the recording load from my master backend leaving only IO management (and various backend duties, and recording scheduling, and mysql database) to the poor little server. I figured four tuners would be adequete and six more than enough. So I bought three PVR-500s. Two went into the master backend and one into the shuttle, mostly for live-tv watching. I doubt we will ever consume all six tuners all the time, but it only took a month before all six were active for a few hours! Overall, six is enough.

As it stands today

After 18 months of using MythTv (started with v0.16), my setup has not changed much.

The backend is still the same almost-overloaded server:

  • 1.6 GHz AMD Processor
  • 1 GB Memory
  • 2 160 GB disks striped for write-performance
  • 2 Hauppauge PVR-500s (4 tuners) doing hardware encoding

The frontend quietly keeps pace with everything:

  • 1.2 GHz AMD Processor
  • 512 MB Memory
  • 40 GB disk for live tv buffering
  • A single PVR-500 (2 tuners) for extra recording power and watching Live TV

Other machines in the MythTV network include my desktop doing commercial flagging and a dedicated mysql server.

Next time

  • Jumpy, jittery, convoluted video and audio
  • ivtv0 warning: ENC Stream 0 OVERFLOW #981: Stealing a Buffer, 512 currently allocated
  • Video lag/freezing on the frontend
  • Large databases’ impact on the backend
  • Old settings clutter the database

Things Adrienne & Dean Do, Think and Write