Tape 010 - all

So guess uh the big discussion came in January uh in general uh thered been some talk
back and forth about things like this over number of months and all of sudden word
came down that management was really seriously thinking of doing this and they wanted to
know is there any reason why we couldnt do it Uh and they also had suggestion that
they wanted to release it in the first quarter which is by 3-31 uh hence the name of this
ongoing project Uh it was uh-uh the big question was what-wotthbtop-us what
was the reason why we couldnt do this And they gave us think almost somewhere
between three days and week to try and decide if there was something that we were not
thinking about and uh what they could do you know how we could do it how
about getting involved and finally said definitely were going to do this mean
fundamentally whats going on is uh you know Microsoft is uh competing very strongly
against us and uh guess this-the idea is to try and roll the dice little bit andmake this
more interesting for everyone Uh theres some old quotes about uh terrain being the
most critical in combat situation and think were smaller company able to move faster
and the hope is that we can bring this to bear so that Microsoft cant compete as well Fact
Ive really seen positive things come out of this where uh weve been rated as better
browser becaus now were cost competitive But anyway so it-it came out in late January
uh there
wer
Iliad meeting with Bob Lisbon the Senior VP of all of client division
and theyve gtT20 other people saying Are there any show stoppers Are there any
show stoppers No one could really think of uh a-a dead problem and then uh guess
Mark Mark Andreeson went public with this that we were definitely going to do it
And then the question is how can we do it quickly Now 3-31 is coming up pretty fast
The decision was made mid to late January so this gave us little over two months and the
tmth is if you back off the schedule and you say well if you want to be done on 3-31 you
really need to be aim for 3-15 so that youre dead youre dead done and then you have
some slack to track care ofrSsizlasruokuattwuds.AtQ1ti
Ahem the goal then is really to get done by 3-15 We had really two months Two
.0Et7
we
Page
Netscape Documentary
Tape 10 Jim Roskind Interview ci
months is an incredibly short piojt cycle In some sense its not product In another
sense it really is Were gonna show the code to people Uh and so there-theres the first
that people were nervous about is all sorts of profanity that might sit in the code Anything Kiu
uh-uh people didnt like certain variant of UNIX and uh there sort of abusive terms
relating to the operating system and-and truth be known lot of us consider this non
professional mean uh theres there are examples where companies have had things
appear accidentally on their disk profanity which is even outside of the code and gotten all
77 r--
sorts of flaks-flak from different groups you know its not part of thcir able code so
were very concerned about that There was concern that were going to be infringing onon
copyrights or trademar-ar-or any sort of restricted code The thing is its hard in this
business to know whats patented Patent things things are moving along very quickly
uh theres certain chance that were using something and dont even realize it People
from the outside cant-cant see what theyre doing what were doing but once you look
at the code you go oh theyre using this algo-rhythm They are using this-this
technique But for engineer it would very difficult to see what source code it would
very straight forward to see it not that we stole it but simply independent
invention and patent its moving along its a-its very scary business at this point So
ub we started looking at that The big thing that hit us that we didnt really forsee in that
first week is the complete consideration of third pirty cride It turns out the numbers are in
excess of about..ttydifferent third parties that contributed portions of code to the system
that we consider communicator And now the realization was gee we cant give away this
third party code without their agreement So now there are-there are several possibilities..
one is to get them to sign up to the actual uh they can call it NPL Netscape public lisense
and try and convince them to go for it Uh but you know in truth alot of these people_are
making money by selling this code mean there are-there are versions of code where you
\pay thousand dollars you get whole big library including source code And you
bundle it into your product and your liense saysyoure allowed to see and use the source
Page2
-1
Netscape Documentary
Tape 10 Jim Roskind Interview
code and ship the binary Thats common scenario Uh as an example Microsoft has
all sorts things built into their compiler uh that are standard routine such as sorting
routine of sorts uh Quick Sort And be it there are no unstandard algorhythms but
theyll actually give you copy of their source But the intent is not that you will
redistribute that source and make your own compiler using their system but rather youll
just use it as component of the system youre shipping So we started talking to
.JesV Adifferent
players One of my jobs historically here has been the job the job-of security
architect obialittl bit the liason through J-Java for source 3-31 one as we call it Um
and so had to consider the issue of Java and approach Java Soft Am running on too
long
So uh so laison to Java had to raise the elements of what can we do about Java Java
represents large amount of source which is bundled into the communicator and is clearly
restricted in source form by Java Soft The question is can we ship it What-Whats the
scenario under which we can ship it And can talk about that if youd like uh but it
fundamentally going to all these different vendors Maybe Java Soft gives good
examples of-of ways that we tried to consider working around the system Uh Java Soft
actually distributes their source under fairly uh easy to get lisense The one restriction is
when you go to their site you have to agree to take it for non-commercial use If you want
commercial use you have to you pay you know dollars and you-you get certain rights to
use it But noncommercial use people get to see it So the approached that another large
uh software distribution or free software distribution system Ticklinex is that they
distribute patch that is list of changes from the freely distributable source Now the
idea is linex gives you this and you get from Java Soft the actual underlying source
Then there are tools that just merge them together and now you have really home-brewed
version So the idea is we dont really ship the restricted source we say go to Java
Soft and get it Well just shif-ship patch But everyone and their brother knowsthe
Page
Netscape Documentary
Tape 10 Jim Roskind Interview
intent is just put them all together In our case we made so many changes to Java Softsvirtual
machineJava Softs source code that it was hard to use tool to do this So the
approach that took is that we went to Java Soft and we said hey could we ship your
source with the changeweve made under your highly restrictive lisense Now it doesnt
help the outside world them much but at least it gives them something to play with it
gives them model \Yeu harc to want to develop a-an alternate YM an alternate uh Java
virtual machine They can plug maybe Cava theres whole bunch of different
Obi
variations Also other vendors can think about integrating it All VM vendors that are
concerned about doing integration already have lisenses can say oh this is how we plug
it in and theyll have much easier time integrating So it would nice for everybody
although it wouldnt be the perfect scenario it would be pretty good Uh the problem is
that with two months to go from basically 1-15 to 3-15 we have to put deadline of
listen youve got to agree to this weve got to get it in writing if were gonna move
ahead If not we just have to pull the plug and say Its out We have to tell the public
and thats exactly what happened We passed 2-24 February 24th that was the official
deadline and at that point uh we can no longer wait for agreement from Java Softwace
We really were gonna be put in position where we couldnt get out of- get out of this
trouble uh so at that goint we began cutting So weve been cutting now roughly 2-24 2-
____ 26 that week iIpuk1 Java out And this represents big effort to not just remove it but
you want to gracefully remove it You want move Java but still leave navigator
functioning That sounds like saying okay we decided we dont want to side window
Uh and maybe you have to now remove it but you really dont air blowing out so you
have to figure out how to plaster that up so that uh this thing is someway useable Yes
you cant look out the window You cant do everything you could do with the window
c4
but it still functions like so its an interesting thing removing pieces and trying to leave
functional body underneath SIn some cases weve had harder troubles with this uh theres
database that we were using its part of mail system and in that case not only couldnt
/0
3S
Page
Netscape Documentary
Tape 10 Jim Roskind Interview
we ship it and couldnt really get all the stuff straight but this was fairly pivotal to all of
our mail side of this product and so the bottom line is we ended up think the current plan
is still lithe bit up in the air is is ...eres little fear that this will be viewed as
misrepresentation to the net but my answer is you do the best you can you tell the net
what youre doing you try to be honest an open and thats think what were going for
right now Youll see that number of different pieces theyll be funny sections uh
security is another example of major piece of code Now this is probably the most
interesting one because the third party we unintelligble all the code but theres this
ery
restriction by the Federal Government on how we can distribute this code its getThe
fekb4t4code So now the hard part is what can we do to satisfy the government whij\
effectively is the third party and ship it Um and this is even stickier because now were
not at risk of being sued were at risk of being put in prison which we consider bad
thing it would slow our software development significantly So anyway the hard
question we put to the government is how much do we have to cut and their answer is
pretty simple they said you have to make it look like there never was any security in the
system you have to make sure theres no sign of crypto or photography encription if you
can do that then you can ship it and now the bad news is cryptography is hidden its not
hidden its tied deeply into the system its almost like with car saying you can ship it
but you cant use electricity in your car Now w-ll csll you the lightbulbs oh no you
dont need lighting the cars only driven by day thats right but the wires you cant
leave the wires in because it shows how you would wire the car and so theres very
large effort think this was Jeff Wienstein and hes put in quite an effort to try and
remove this thing which is very integral in fact initially one of the major components of
navigator one of the things that differentiated it from related products think Mosaic was
that we had SSL secure socket layer this is cryptographically strong way of talking to
sites Alot of commerce feels much more comfortable banking and buying feels much
more comformtable communicating across this but this uses cryptography uh and the
Pages
Netscape Documentary
Tape 10 Jim Roskind Interview
bottom line is we couldnt leave that in at all People started to wonder well gee heard
you can export weak cryptography but not strong cryptography why wont they allow you
to export source for weak cryptography the answer is typically the difference between
ti
weak and strong is whether you type 40 or 128 at oneoint which is very visible So
theres no real way to narrow it down This is an example of the same idea of cutting
things out but the goal is to make it useable so now comes question what happens
when you go to site that expects you to use SSL how does the system respond and is it
exceptable And so the cutting has to be very delicate Im sure you can imagine if you
had no electricity all of sudden youdf have to think you know how do we fire the
chambers well OK its diesel engine thats right we use diesel engine no electronic
ignition no headlights uh we cant have warning lights on it so all sorts of things have to
be 44d out and the end result is yes you do have car it wouldnt surpirse me if some
early cars had very little in the way of electrics or could be managed in that way but its
alot of work to get this done And so theres you_see java going away you see crypto
going away uh in the end some people will come and say hey why did you say you were
shipping this and were hoping that the Net will be understanding of some of this uh we
cant its
interesting with the crypto laws we cant even leave holes for plugging in crypto
it has to just be completely unclear where the crypto goes in There are laws that youre not
allowed to ship whats called sort of plug and play crypto you cant even make it so oh
this is where you would plug in crypto module but were not allowed to plug it in It
has to be completely devoid of connection so thats its very difficult this is all theres
uhole lot of arguing about whether this is right or wrong it certainly impacts us but in this
case you have too months to go you cant go about this in any slow way you have to be
very direct the most direct ways complete removal
Describe the process of code writing
rz. /0 /7
iY/
ii
Page
Netscape Documentary
Tape 10 Jim Roskind Interview
There is some code creation in the analogy said you remove window and you have to
put in something so that the air doesnt blow out and people see find intenable untenable
so really pain to driving car uh so you have to put in wooden plank in the window
uh now that has to connect to the other pieces like there used to be crank to crank this up
and down you have to tie that in you have to go in and make delicate adjustment the bad
news is while youre doing this someone else may decide oh that crank is really third
party uh software you have to replace that crank with uh uh an electric power
window just to get by no more cranking up and down just locking so theyre chaning
the lock youre changing the window and when you go to connect you each look for the
bolts you know they guy changing the crank looks for the part of the window and goes
wow its gone the guy putting in the window looks for the crank and goes theres nothing
to connect with this issue of when you have many people working on the project at the
same time and really rushing you cant just say OK you replaced the window OK
youre done and now that youre done OK you replace the crank People are trying to
modify this large code base think its in the neighborhood of million lines of code and
theyre running around we have Imnot sure what the count of engineers is certainly Idl
say theres in excess of 100 engineers involved in this in various ways theyre all running
around with different tasks and then they have whats called merging problem basically
youve go off with youre snapshot design of the car which is the code the code is really
you know design and you come back with the changes that youve made with your
design and youre about to check it in and make it officially part of the design and you
notice right where it checks in that the picture has been changed and now you have to
merge you have to carefully look at their changes and your changes find way to work
them together Now when youre working on veryjiow schedule or if its just one
person everything is nice and sequential everyone knows whats going on Uh when
youre working with 100 people its very very difficult lets see if can site an
example of merge problems Uh here am Im working on group withftom Ronsi te
ii
Page
iC
Netscape Documentary
Tape 10 Jim Roskind Interview
andUbert Shaw and were tearing out java Well java it turns out makes alot of security
calls and now the question is what should we do about those security calls Well the
security group is assigned to go in and cut out all the security so they actually went into
Java and put little cut marks around that At the same time we said well were not going
to call anymore so ptI lets just adjust this piece of code so that it doesnt make the call
no we go to chcck it in and suddenly we realize they changed code we changed Now in
the case of java were about to check in
think its in the range of 50 files and very large
number of changes and now were about to check it in and it gets pretty scarry knowing
that its about to break if it gets put in Now if you break the build thats the technical
phrase if you put in something that doesnt allow complete car to be rolled off see
everyday they take snapshop from that design they build the car and they test drive it
So if ever its the case that car coming off the assembly line doesnt work and they keep
testing them time and again its this as this design evolves and they just stop and
everyone panicks little and its usually not even clear whats broken all they know is
Imin the car put my key in the ignition turned it and nothing happened And well
the answer is if you took away all the electricity you now have to crank the car thats the
new way and the guy says oh didnt even realize yeu had to crank the car and jill
maybe the guy putting in the crank connected to where the starting motor goes but the
whole starting motor housing is gone You know so theres all these miscommunications
and so the problem is that once the system falls on the floor everyone gets scared And
the goal is to stop changing anything uh the hard part in debugging or finding the problem
is to locate the actual problem In this ease if$ have very short span of time were
sure theres only been couple of changes and we know to look at exactly those changes
If have long span of time it gets very cousing
as to which one of the many changes
really is what broke the system So typical1 once the system breaks we just stop dead
and we try
to then do little autopsy and look back so we can see whats chang and
what we can fix So uh the idea is that we want to cause everything to stop but the bad
iS
Page
Netscape Documentary
Tape 10 Jim Roskind Interview
news is everyone
else is racing to put their changes in theyve made their changes they
want to put them in and then go on to the next task And so now when the whole system
stops suddenly you have this is this assembly line it all comes to halt and now you
have people spinning their wheels saying well want to check in my new side view
mirror uh but the whole door is apart cant really see if this new side view mirror will
work and4f-ycu put it down then can go work on the hub caps but then the bad news
Im going to forget what was doing and Ill just wait Ill jsut wait for half day and
get this stuff checked in But now what youve done is youve instigated inefficiency
the system Again two months its desperate schedule you cant have any ineffeciency7
so the typical deal is when you go to check in if you see that theres going to be confIii
rather than risking breaking the system you have to slow yourself down carefully poll do
the integration be sure now you have perfect snapshot of the system that really worked
and finally say ah now Im done and then you turn around face the system and go check
it again and much to your chagrin something lese has changes and theres another piece
that has to be merged Yoücbuld actually call this merge hell You were mentioning this
in some of the meetings different people are in merge hell basically the time that this
occurs if you have very large change that really stretches across alot of the system then
its almost guarenteed to hit someone else And everytime you fix one thing but you wait
two hours four hours day youre guarenteed something else will break So finally the
call whats called the car pool and they say listen youre in merge hell we can get you out
stop changing the system the world stops people stop checking in its not broken its all
working you get to poll calmly take deep breath fix all the differences check your
drawing change in and then release re-open the tree will be the phrase So then youll see
this commonly happen when people get
into merge hell theyll stop the tree from
changing theyll allow merge they call the car pool car pool its really easy to land
small change if all you want to do is crss off the DOT sticker you know on the window
you can go in and do that and be done And then you go to another window and do
Page
Netscape Documentary
Tape 10 Jim Roskind Interview
another DOT If you have to remove all the glass in the car at once and it turns out that
the glass is related to other things electricity thats an easier example you cant jsut say
well Ill remove the battery so then the car stops working You have both remove the
battery remove the starter insert crank you have to remove the ignition switch you
have to insert all these things have to be done at once you cant get away with doing one
without the other And so thats why you have this large change Its always nicer some
people will try to break 1arget theyll say well could take out the lights the
electricity has to go Ill take out the lights first OK thats out Now can remove this
other piece But at some point you get down to the starter motor and the ignition system
CoCV r4pC
and it all has to be changed out at once you have to switch fro ignition system to
diesel and you cant do it piece meal and theres lot of other people doing skip checks
Yesterday
Yesterday was Monday the start of the week and people were getting really nervous
cause now its 3-10 and theyre hoping to be done really by 3-15 basically by Friday
and there are two major car pools that hadnt landed One is the security car pool and the
second is the java car pool Now there was sort of race between the two of us to see
who can get out there fat faster or be ready faster there was maybe almost there was
another thing we talk about here called schedule chicken uh schedule chicken is where uh
one group realizes maybe my group could realize we couldnt be done by Thursday but
we tell them we could be done by Wednesday knowing that the other group wont be done
by Friday and all the attention will be focused on them and then we can be done by
Thursday and no one ever knew that we were really ever going to be late So its game of
chicken the question is whos going to tac cnt7 CCn In
Page 10
Netscape Documentary
Tape 10 Jim Roskind Interview
..tape begins and finally yesterday at the meeting they announced they were going to
check it uh and it was going to happen in the afternoon the tree was being closed so that
also meant that since they were going to check in the tree was going to stop moving and
certainly a14 call p001/ now put on the back burner They told us at the time at the meeting
worse case it could be done by Wednesday which is scary thought because again Friday
cGr
is the goal we have major eØl pool coming in were all so
scaredk
we dont want to be
on the hot seat we dont want to be the one to cause the prob the program to slip uh so
car
yesterday there was alot of discussion about the landing the security call pool and what
what is...exactly is uh the uh the Java lIpool gonna do about this the first thing we
mentioned we were then now were talking about recovery plans mentioned we took
snapshots of where things were before the security Sipool started landing in their change
caused too much bustage again another clever technical term uh the-the chanebes we
could always roll them back out and let us go first and let them think about what they were
e4 Iccrd
going to do little bit more clearly Uh Ynfle mentionedfl Jeff Weinstein or his
people coming from his group his goal he was just going to stay all night and be sure the
call pool cleared everytime there was an error he was going to turn it back around The
tree wasnt going to be moving so the ground wasnt going to be shaking under his feet
and he was going to get it all done The good news is actually think by about Im not
sure if it was or 11 oclock at night he actually was completely done So then we got
call was here Rahman and Yube and uh Hey their call pool cleared how are you
ri
guys feeling whether we lockcditWtui1aiiitnrbuitthng- not at that point Its little
scary going in late at night when whole bunch of changes have landed because you dont
have the full support of the other engineering staff if there was problem outside your urea
e4 expertise its
pretty problematic Youll see today in meetings theyll be talking about cc
the Java call pool.. Carnrnnland is it going to make it What exactly are the risks
involved Last night we did merge the tip which is trying to adjust our design to be as
LC
Page 11
Netscape Documentary
Tape 10 Jim Roskind Interview
close as possible to the most recent design most recent set of code and uh it merged up ç.j i3
nicely but then when we brought it up all of sudden we couldnt it called clicking on
links you know major function you bring up the whoeage you see an underlined word
it talks about some interesting part tbati-4terpiiig4ttllpprttugetber and it just
sort of stares you in the face and go holy cow somethings brokenl and now came
question.. did someone else break the system before we entered this new sytem status or
did we just break it when our merges conflicted with items that were sitting there So we
were trying probably til about three in the morning to figure out what the heck is going
on finally decided the answer is take break go home get some sleep come back in
today Tuesday uh have some more people with more expertise in Unix to look at the exact
Unix problem were facing and uh see if we can diagnose it So were little bit on the
hot seat We thought we were good to go were about ready to check in were doing
final final final checks making sure everything merges again We cant afford to land
unless we can assure safe landing Uh and now we suddenly had a-a problem Were
going around were circling the clock is ticking its Tuesday theyre going to wonder
are we going to make it Theyre on the outside theyre listening to me and they have to
wonder. is Jim really understanding whats going on are these guys misrepresenting
intentionally Es really common theres an old joke uh in uh computer software where
they say you know are you gonna be done and the answers oh this is the last bug
here This is the very last bug.t And obviously every time you get one more bug more
things work and then often you find another bug But each time you find one everything
up to there ran perfectly so you have this funny feeling that everythings working
wonderfully without there being problem in the design uh and so its pretty nervewracking
for management Management commonly has this problem with software
engineering that its-theres alot of stuff going on its easy to misunderstand details uh
you know its-the-anothe/guery is that ninety percent of the work is finished in the last ten
percent of the time This is true of building house Its true of building so many things
Page 12
Netscape Documentary
Tape 10 Jim Roskind Interview
and uh it gets very confusing People want to be done they dont want to be in the hot
seat they think theyre done but they dont have good way to know for sure that theyre
dead done So uh Im pretty confident have pretty good team think were
representing it well but youll see little bit of termoil Youll see alot of nervousness in
Michael Uh hes driving this uh ovei time theres rapport and theres confidence
thats built up so when tell him that were in good shape hes going to tend to believe
me But at the same time the clock is ticking and you have to see what 11 happen So
thats-thats the big stuff thats to be going out today alot of worries about that We do
open the tree up to the call pool so alot of little people got through alot of little items
these are not little people people got to inject small changes last night and so that released
some to the tension note that if you close the tree to long for call pool you slowly get
ii cr
more and more people standing up and shouting theyre being slowed down Soa-eah
pool is meant to expedite this one person but it does it does it at the expense of group
and you constrantly have to trade off if you slow the group too much and you finally have
to get out of the way and let them go by So its this interesting delicate balance and the
constant question Michael is wondering. are they being honest with me Can they
get this done in just few hours Or should let the people who are stacked up behind
them go And he has to mak at judgement call And uh guess thats why they pay
him some money
Uh you can do that Right now uh oficially right now have three one-on-one
discussions a-aside from the fact that have a-unintelligible first check to see if theyre in
yet They havent come in
yet thinls
unless unintelligible
Oh yes yeah totally
Page 13
Netscape Documentary
Tape 10 Jim Roskind Interview
Uh oh the answer is.. most of the stuff yet-got most the stuff is everyone in thc room
going why is this breaking sheet whats going on here Last night you know we are
looking why is the suddenly breaking and theyre going goshw-w-w- i-iits It
fç
doesnt thet that you see is people working Theres the three of us on Tuesday
p-nti
nights and Thursday nights have been trying to go out and play raquet ball -.hfec.ea1lLL
And this is now looking like unless we get this check in theres no raquet ball tonight and
uh so it will be interesting in general try you have to realize also as us managerse
callid pointy-head people its take off on certain comic strips that represent managers that
way To be good manager you really dont want to come and look over guys shoulder
every two seconds and say Hey are you done yet eally doesnt help them It
adds to the tension Uh the truth is my job is to try to shield the pressure so go to the
meetings and Im going to sit through and take the flack about us not being in and us not
being ready explain whats going on and allow them to focus as much as possible on
getting the job done So thats-thats my job and have to sort of weigh this delicately and
be nice guy as come around asking without interrupting being curteous to system In
fact you know my job bring cookies bring water Uh if they have problems and they
want me Ill sit down and look at code with them as well uh It varies so its just
moving along wthei4-ache.4 and time you never know when you have the last bug
You never know when the last thing on the main system is about hope it all works And
VS
the truth is if youre doing good it really does time again-is immemorial here
Thats it And it really does work Okay that was the problem and were good to go but
you dont quite know this and its uh you get good reinforcement when it happens then
you hit your head on another one and youre going Oh darn its not over you know
the marathon hasnt ended thought could see the-the light at the end of the tunnel but
it..
Page 14