Sunday, December 28, 2008

Carrots, sticks and the coding class

Friday the 26th was the last working day before the winter holiday which will end on the 5th of January when we'll all go back to the office.

The 26th also marked a milestone by which our project had to reach a certain number of features and run all of them at a certain frame rate.
Being currently on PC, on hardware that is not the final platform, it's hard to define quality versus frame-rate.

The 23rd was a national holiday (Emperor's birthday). I took the 24th off to stay with my girlfriend, as Christmas eve is really more of a couple's event in Japan and on average I see my gf at 1 AM when I'm back home for dinner... The 25th I was back in the office.. working into 26th morning for the milestone. I then went back home, slept a couple of hours, took a shower and showed up in the office by 11 AM for the deadline.

The deadline went ok. The features were in. Loading somehow sucked on that machine, but framerate was also generally on target (sometimes faster sometimes slower). We got the OK from the bosses, and that was it. ..or so I thought.

Fast forward to the last meeting. We show the same demo to another team related to the project.. it's a Friday evening before the holidays. Everybody pretty much with a foot outside the door.. but my boss stalls.. he does not say "OK we are done". He stalls and then comes out with some request for more quality in trade of framerate (completely voiding the goal for which we all worked for the past week or so).. and he wants that for the 5th, the first day of work after the holidays. Did anyone just say "work on your holidays" ?!!
The plan is that someone works his ass off for a couple of days during the holiday (that is always me, or me plus a couple of coworkers) and then when we all go back, we get a nice pat on our back, the famous "carrot".

I say "OK", I can increase texture resolution, whatever, I can do anything. I answer because of my pride, but also because I don't want to argue.. it's pointless.
The meeting left everyone shaking their heads in disbelief. My boss could claim that he's just trying to motivate us, or that he's taking the heat in order to bring the best out of us.. anything can be said, but I honestly really have to call BS on this one.

I spent the whole night into the morning of the 27th to get up a simple system that makes it easy to choose rendering quality vs speed. I had to add some sort of console with some options and I also took the opportunity to more or less consolidate the application's states/options with a system that can save out and read in the states in a file.

What I really had to do is not the point. The point is that on the last day of work before a long holiday, a boss should not suddenly put stress on his workers. That is not work, that's attention seeking.
Maybe he didn't realize what he was doing, maybe he has a higher calling.. whatever that was, it was BS.

Being a manager is not easy. Management is easy to blame and people will often do that just as a sport. So, I try to be understanding... but all those little things have been accumulating for a while now, and I'm starting to lose my patience.
Myself and the people around me have been doing a terrific job. We are putting a lot of effort and I think we are making a sensible difference for the company.. so, no more carrot & stick, we are not kids anymore, just let us work.

Happy holidays !


  1. That sucks,

    Sometimes 'bosses' make those last minute demands just to show that they are involved in the process, despite the fact that it pretty much voids the direction of any prior efforts.

    So when are you getting the final hardware?

    BTW my wife got me the 160GB PS3 for Xmas, it came with 'Drake Fortune', so I spent a couple of hours playing a computer game!!! (its been a while).

    Then I decided to see how easy would it be to program on it copared to programming on the actual devkit that I have at work, so I partitioned the HD to set aside 10GB for 'other OS' and installed Linux Fedora 9 distribution on it. I've never used Linux before so it was a bit of a challenge. For example being able to log in as root wasnt so obvious to do.

    Fedora has a windows based GUI, but it runs kind of slow on the PS3, my guess is that memory is the bottleneck. I also installed the Cell SDK that is available from IBM, which came with a Cell based IDE, "Eclipse". It had a project wizard that allowed me to setup a basic framework for a project with PPE and SPE modules.
    But when I tried to build it, I got errors saying that it could not find some system paths. Thats pretty much where I left it.

    Another problem is that its is much easier and faster to use the PC to browse any Cell development docs, so I have my PC and PS3 connected to the same monitor, where the PS3 is using the HDMI input and have to constantly toggle between the 2. I believe there is a way to connect to the PS3 using remote desktop, which should make this process a lot less annoying. Thats what I am going to try next.

    However the even biggest problem is finding time to do all this...

  2. I'm not sure about the final hardware, we are getting closer. I think it will be on an i7.
    More than that, I'd like to have more time to profile and optimize. There is a lot of processing going. Especially scary all the "streaming".. which is actually really just loading in a separate thread.
    Recently I introduced shader compilation caching.. we have 851 shaders so far, and compiling all of them while loading was taking some time.
    Eventually we'll compile in the build phase, but for now it's good enough to have the runtime compile in a cache dir the first time.

    I've been thinking of buying a console myself, but I'm afraid of getting addicted 8)
    I have a 360 already, but I've heard that PS3 can be used as open media player.

    How about getting a second monitor ? It's very important to develop on a PC alone, imagine on 2 separate systems 8)
    As for the remote desktop, VNC is pretty much anywhere, maybe there already is some package for that Linux distribution..

  3. I think the overhead for dynamic branching in shaders is not so bad anymore, so maybe you can try and combine some of them together to reduce the shader explosion?

    Second monitor would be nice, but there is still no way to cross compile or remote debug the PS3 from a PC.

    So I dont see how a second monitor would be better than using a remote desktop. I will still want to browse the net and read documents on the PC so I think it would be easier to have everything controlled from the PC as opposed to having the PS3 as a totally independent system with its own keyboard, mouse and monitor.

    VNC is what I am planning on using, the process of setting it up is already documented by a programmer from Insomniac games.

    Unfortunately the Cell SDK doesnt give you any access to the GFX card...

  4. I guess I meant that you use one monitor just for the PS3 output and the other one for the PC. But, definitely, better to use VNC if then you need two keyboards and 2 mouses.

    As for the lack of access to the GFX card.. maybe a good opportunity to get some software rendering going again ;)
    It won't perform as well, but at least it would be something different.. but that really depends on what one is trying to do.

    Regarding the shaders.. they are actually automatically generated by the TA team. We haven't yet sat down to think on how to optimize the thing... and, most importantly, not yet figured the performance penalty related to have so many shaders. It may end up not being so much of an issue when it comes to rendering.

  5. I assumed that the VNC viewer actually displays an exact copy of the PS3 output, but after finally getting it to work, I see that it is not the case, so you are right about the 2nd monitor.

    Software rendering is what I was going to try to do eventually. Parallelizing the work across the available SPE units.

    But the 1st step is to get a PPU module to load an SPU program and get some data going back and forth. Looks like there are some samples and online tutorials to follow.

    I am going to use the 'Eclipse' IDE that is included with the Cell 3.1 SDK. Looks pretty nice at 1st glance although it is Java based.

    Regarding shaders, the more specialized they are the better the performance, but then the number of shaders grows. If you can combine some effects into 1 shader and choose between them with some variables at runtime then you sacrifice some performance but they become easier to manage...

  6. Stage 1 success!!!

    Built a PPU program with an embedded SPU executable that uploads the SPU code and executes it:

    Note the 8 available SPUs. Typically we can only use 6 in a PS3 game, because 2 are taken up by the PS3 OS, but since I am running Linux all 8 SPUs are available!!!

  7. Woo.. looking good !

    Weren't the SPUs limited to 6 because also one might be broken ?

    Only the code indentation is a bit gay, but I guess that's the same code 8)

  8. Paul, is ... is that Blender in the background?! Nice ... Blender's my "3D Editor of choice" at home. :D

    Mr. Hard-worker Kaz,

    I'm sorry to read about this thing at your place of work. I've tasted the "get it running at high frame rates on non-target hardware" experience and discovered it's really a waste of time and don't really understand why your boss decided to "eat" your vacation time (which you are legally entitled to); life's CPU cycles are too short to burn in this fashion.

    I hope things "normalize" over there ...

  9. Not to mention the two productivity tools Emule and uTorrent.

    I can understand the latter because a lot of Linux distros are distributed as torrents, but emule? :)

  10. I think you could not assume 8 working SPUs and have to do some querying. I must be lucky that my PS3 has all of them working.

    The code indentation is not mine, it was a sample.

    The challenge was really getting it all to the point of building and running.

    I tried Blender, but the UI and controls are somewhat weird, so I have it, but use Maya.

    And emule gets you what you want, slowly but surely.

  11. Ragin,
    It's funny because a couple of weeks ago we had some big meeting where we were told not to overwork unless explicitly asked for. Kind of a joke because having a ton of work is just like being asked to overwork 8)
    But I didn't complain. Only this time it's ridiculous how it came out that we basically had to do (some) work over the holiday, by just requesting a feature, without saying "please come" ;)
    It's not big deal, because it wasn't an actual imposition and I could have specifically complained right there (though one doesn't really want to be a complainer or a quitter) .. still, it's funny.

    So your PS3 is of a good breed !!

  12. Maybe it was an end-of-the-year joke of some kind. 8P

  13. can't believe you're complaining about your last minute boss' requests!

    it's exactly the same for me and almost all the other workers, but you're developing videogames and high quality real time rendering engines, while I'm handling (ie. not developing while I'd really like to) boring slow fat unreasonable complex relational data and related ugly incoherent software...

  14. In the end we all live in a Dilbert world 8)

    My current job may be more stimulating but I easily work 13-14 hours per day. The fact that the job is interesting becomes a problem, because then it's harder to get pay raises or claim vacation time.