No. But I am owed a beer for every project I show up to. Happens around twice per year.
Before a project that involved outfitting a survey ship headed for the Mediterranean, this conversation took place: (paraphrasing)
Me: This storage cluster is based on 10gig/s network. I don’t think the data processors will get the throughput they require.
Chief tech: I tallied up the data transfers, and the time available should suffice with 10g/s.
Me: Did you account for the fact that they duplicate the data and transfer a lot of it twice?
Chief tech: No. But I’ll tell them not to do that.
Some time goes by. Project starts, 1PB of data is collected, and processing is starting up, scheduled to complete everything shortly before they arrive back in the home port. Then my phone rings.
Project manager: Throughput speeds are too slow. Is there anything that can be done, or do we have to eat the daily fine for being too late?
Me: How much is the daily fine?
PM: (number) USD
Me: I’ll make the purchase req now for one tenth of the daily sum. That should increase the throuput.
PM: Hardware install involved?
Me: Yes. I’ll order, configure, and document everything for the chief tech to plug&play during their refuel stop.
PM: I owe you a beer for the next projects.
The extra cost wasn’t that much, considering it was the same upgrade I recommended before project startup anyway. We just had to pay for express freight from the vendor to my home, and from my home down to the ship.
Long story short, it worked out. A Mellanox 2100 was shipped down to Malta, and installed by the crew, and it worked out of the box. I was worried about the extensive VLAN and LACP setup in play. But as that setup was my design, I managed to wing it when preparing this switch. I had been using the production cluster for some testing earlier, so it already had the necessary kernel modules for the 100gig cards. The processors had everything wrapped up and written to tape the day before arriving back home. The night of arrival there was a project completion dinner for the crew and us field service personnel. I drank for free that night. I don’t remember it all, except that it was fun, and that the only reason why I called it a night at 0300 was because the hotel bar was closed.
To the chief techs defense, he wasn’t that wrong. He’d made a reasonable estimate based on the numbers he had available. Unfortunately he didn’t have all the numbers, as well as give the processors some head room for fudge factor. This chief tech was mostly experienced in only providing raw dataand thus working without the processing aspect.
No. But I am owed a beer for every project I show up to. Happens around twice per year.
Before a project that involved outfitting a survey ship headed for the Mediterranean, this conversation took place: (paraphrasing)
Me: This storage cluster is based on 10gig/s network. I don’t think the data processors will get the throughput they require.
Chief tech: I tallied up the data transfers, and the time available should suffice with 10g/s.
Me: Did you account for the fact that they duplicate the data and transfer a lot of it twice?
Chief tech: No. But I’ll tell them not to do that.
Some time goes by. Project starts, 1PB of data is collected, and processing is starting up, scheduled to complete everything shortly before they arrive back in the home port. Then my phone rings.
Project manager: Throughput speeds are too slow. Is there anything that can be done, or do we have to eat the daily fine for being too late?
Me: How much is the daily fine?
PM: (number) USD
Me: I’ll make the purchase req now for one tenth of the daily sum. That should increase the throuput.
PM: Hardware install involved?
Me: Yes. I’ll order, configure, and document everything for the chief tech to plug&play during their refuel stop.
PM: I owe you a beer for the next projects.
The extra cost wasn’t that much, considering it was the same upgrade I recommended before project startup anyway. We just had to pay for express freight from the vendor to my home, and from my home down to the ship. Long story short, it worked out. A Mellanox 2100 was shipped down to Malta, and installed by the crew, and it worked out of the box. I was worried about the extensive VLAN and LACP setup in play. But as that setup was my design, I managed to wing it when preparing this switch. I had been using the production cluster for some testing earlier, so it already had the necessary kernel modules for the 100gig cards. The processors had everything wrapped up and written to tape the day before arriving back home. The night of arrival there was a project completion dinner for the crew and us field service personnel. I drank for free that night. I don’t remember it all, except that it was fun, and that the only reason why I called it a night at 0300 was because the hotel bar was closed.
To the chief techs defense, he wasn’t that wrong. He’d made a reasonable estimate based on the numbers he had available. Unfortunately he didn’t have all the numbers, as well as give the processors some head room for fudge factor. This chief tech was mostly experienced in only providing raw dataand thus working without the processing aspect.
Thanks for this story, I enjoyed reading it.
I enjoyed reading this, hopefully you share more stories on Lemmy in the future.
I probably will when relevant