This is the first chapter of «Notes of a Traveling Cat» — stories from two decades of programming in C++ across very different fields. The whole book can be downloaded as a PDF or EPUB (in Russian).
For my children — Katyusha, Matvey and Rita — I hope I showed you the right road…
The trouble, as so often happens in our thankless line of work — keeping systems running in the field — began quietly and unremarkably, almost imperceptibly, like the first cracks in spring ice, when the surface still seems solid but somewhere deep inside the irreversible processes of breaking and thawing have already begun. At a serious facility, one motion sensor failed first, but it was due for replacement in six months anyway, so we wrote it off as chance, as ordinary wear or a power surge — anything could have happened. But then, a few days later, another sensor went; the plant people have standing instructions for that, and the head of security called our director with a faint note of bewilderment in his voice — "What's this all about, then? Your gear was supposed to be reliable, and now they're dropping one after another."
And then, over the following week, three sensors burned out at once at different points along the perimeter, and it became clear that this was no longer chance, no longer coincidence, but a downright "fire drill," and someone had to fly out there at full speed and fix it. Our small firm in those days, in the early 2000s, was in a rather specific line of business — access-control systems for high-security facilities. We were neither especially large nor especially rich, but there weren't all that many firms of our kind in the field, and there obviously weren't enough contracts to go around, so we guarded our reputation closely, because reputation meant future contracts. And the contract with the shipyard in the town of S. was a real breakthrough for us, a fat enough morsel that we'd practically clawed away from a bigger player, and that contract gave us not just a steady living for the next couple of years but the chance to work with serious people at serious facilities.
In those days, when the sensor failures started piling up, the director paced the office corridor quickly, darting from the "physics guys" to the "installers." Sometimes the developers caught it too, and they'd be sent "into the field" as well, because all the teams were already out at facilities, and pulling them off meant patching a threadbare blanket with a single patch. The director promised to sort it out, said it was a temporary glitch, said we were already heading out to the site and everything would be fixed there in the shortest possible time — but even through the closed door of his office you could hear notes creeping into his voice that hadn't been there before.
Reputation matters more than money — he repeated this to us constantly, at every meeting, at every opportunity, like a mantra, like an incantation, and there was truth in those words, because in our business, where everything is built on reputation, on the client's confidence that your system won't let them down at the critical moment, a single failure can wipe out years of successful work. Back then I was an ordinary "engi-nerd," sitting in my corner of the office at a computer poking buttons — that is, writing the server side of the system, fiddling with data-exchange protocols, tuning the processing of signals from the sensors and the volumetric detectors, which reacted to changes in infrared radiation — that is, crudely speaking, to the heat of a human body. Our systems weren't the most elegant from an architectural standpoint, in places even a bit dated and clunky, sometimes thrown together in haste under shortages of time and resources — but they were reliable, proven, working on a simple and understandable scheme: sensor – event, signal – controller – server. The server writes to the log and makes the decision, everything is logged, everything is under control, without any magical black boxes.
That's how it was supposed to be, that's how it was conceived, and that's how it was implemented — but reality sometimes makes its own corrections, and things turned out quite differently. The burned-out sensors, those little plastic boxes with electronic guts inside, were dropping strangely, inexplicably strangely: most often it was the volumetric detectors that failed, the very ones that react to a person's presence in the monitored zone, less often the contact sensors, which are extremely simple by nature and, you'd think, shouldn't break at all — but what is there to break in four wires and a battery? At the meeting before the trip, our "physics guy" — a fellow of about thirty but already with a beard like a priest's, who was responsible for the whole physical side of the system, for the installation, the wiring, the power — came out in favor of voltage spikes in the plant's grid: maybe, he said, some heavy equipment switches on and off there and throws a surge that managed to slip past the line of uninterruptible power supplies and kill the sensitive electronics. Someone muttered about a bad batch — apparently we'd bought "knock-off" gear from the wrong supplier, cut corners where we shouldn't have, and now we were paying for it. Nobody said the word that starts with S out loud, but it hung in the air too, because competitors were installing a second security perimeter right next door, and although it seemed paranoid, when you're working with a high-security facility you couldn't simply dismiss that version.
Two weeks into this growing crisis, when the number of failures had passed a dozen and the head of the plant's security service was already talking in plain terms about terminating the contract and bringing in other contractors, the director summoned me and a colleague — let's call him Max, a developer like me, only older and more experienced — into his office and said it short, without mincing words, looking us straight in the eye: "Sort it out on site, take as long as you need. Go there, see with your own eyes what's happening, check the system from end to end, but find the cause. Or we'll be sorting things out without the contract, without the reputation and, possibly, without jobs, because nobody will want to deal with an outfit that botched the work there."
The road to the north is always a kind of special crossing, almost mystical, from one reality into another, from the bustle of big cities into a stillness and vastness that both soothes and unsettles with its endlessness. The farther the train pulled away from Piter and the more kilometers fell behind, the less there was of that endless urban bustle, of the constant tension of the northern capital, and the more there was of sky — huge, high sky that seemed to grow ever bluer, ever colder as we moved north. The train went slowly, swaying on the rail joints; outside the window the forests thickened, at first mixed, then more and more coniferous, dark, with tall straight pines and firs, between which you could glimpse snowy clearings and frozen streams. Then, somewhere past Vologda, the snow started, even though the calendar said March and in Piter the grey city snow was already melting fast, while here a full-blown winter was under way, a real, northern winter that had no intention of yielding to spring. Max sat across from me, looked out the window and smoked in the vestibule, coming back smelling of tobacco and frosty air, while I leafed through printouts of the logs, trying to find at least some pattern in the failures, some hint of what might be causing them.
In the evening, when it had grown dark outside and the train was already moving through total blackness, occasionally cut by the lights of small stations, we drank tea from glasses in their podstakannik holders and discussed theories. The power-grid version was the simplest — the plant probably had some old Soviet wiring that could pick up interference from heavy equipment, from welding machines or large connected loads, and we'd just need to install better filters and the problem would be solved. I wasn't so sure, and a wiring problem would have shown up immediately, but I had no alternative ideas, so I nodded and agreed, though inside I felt that it wasn't that simple.
The town of S. met us with frost — not vicious, but so clinging that it got through any clothing — and with white snow that made you squint even under a sun hidden behind clouds, snow so clean and untouched that everything around seemed brighter than it really was. The town struck me as strange, not quite ordinary — it stood right at the water's edge, the White Sea nearby, which freezes in winter and turns into a white desert you can walk across on foot, and all around grew pines, tall and straight, and there were small lakes hidden under thick ice. Snow lay everywhere — the way it should lie in the north — deep, serious, with no hint of an early thaw. The buildings in the town were mostly five-storey blocks, the standard Soviet boxes, grey and identical, but when you walk between them you get the feeling they were once built to last — not the Piter Khrushchev-era boxes that shed crumbling cement. Time had added its own shades to them all the same — here a darkened façade, there a creaking balcony — and suddenly, among these identical houses, you'd come across a wooden one with carved window frames, slightly tilted, as though it were the local old-timer here; and it really was an old-timer. The town itself, to be honest, isn't old; it appeared back in the Soviet era, grew not gradually like the northern settlements that stretched for centuries along rivers and sea shores, but almost all at once, by plan, by blueprint, by necessity — and yet the land around it holds a far older memory, that of the Nikolo-Korelsky Monastery, and when you think that the monastery saw the Pomors, the storms, the long winters and the ships that sailed off into cold waters, you begin to understand that the place here is far older than the town itself.
But the town itself was born not of trade and the slow sprawl of streets, but of a task, a spec, solved on tight deadlines, when in the thirties people who knew how to build ships and factories came here, and on the shore of the cold White Sea they began raising shipways, workshops, slipways, and houses for the workers alongside. At first the town had a different name, and there wasn't a hint of romance in that name either, only a statement of what it existed for. And amid all of this stood the plant — enormous, sprawling for several kilometers along the shore, with tall buildings, with chimneys, with floodlights that lit the grounds at night so brightly it was as bright as day. With passes and document checks, without fuss but very attentively. And the people here didn't fuss, didn't hurry, but you could see they answered for their stretch of the work, everyone understood the importance of what went on behind those fences, behind those concrete walls with barbed wire on top. The guard at the checkpoint, a man past fifty with tired eyes, spent a long time matching our photographs to our faces, then phoned our handler from the security service.
The plant smelled of oil, metal shavings and something else, something barely perceptible, some particular smell of large-scale production that can't be put into words but that you recognize at once if you've ever caught it. The head of security met us in his office on the third floor of the administrative building and turned out to be exactly as I'd pictured him from his voice on the phone — grey-haired, lean, a Belomor cigarette in his teeth, a retired military man pushing sixty, with a straight back and a voice without any intonation at all: "Our system worked," he said, not inviting us to sit, looking down at us even though we were about the same height. "Then it stopped working. You installed it — you fix it. You've got a week."
We stayed in the plant's hotel, a small three-storey building next to the plant grounds, where visiting specialists, engineers, military acceptance staff and everyone else with some connection to the plant put up. The rooms were very Soviet, judging by the price tags and stickers left on the furniture, almost ascetic — a bed, a table, a wardrobe, and the facilities at the end of the corridor. They fed you well there, northern style, without frills but filling and with some imagination: fresh fish, probably caught right there in the White Sea, meat with braised cabbage and raisins, black bread of a kind no longer baked in Piter, heavy, with a crackling crust. The canteen ran on a schedule, and you had to show up for breakfast before eight in the morning, which for people used to turning up at the office closer to noon felt like a small but real torture.
In the evening, coming back to the hotel, you realize how very quiet it is all around, and you can hear the crunch of snow under your feet, that characteristic, dense sound that only happens in hard frost, when the snow is dry and packed — there's no such silence in big cities, and even if you move out to the suburbs you'll still hear them humming. Above the plant, floodlights burned in the dark sky, coning the swirling snowflakes, and the sight was at once beautiful and palpably cold, getting in behind your collar and driving you back to your room.
The system we'd installed at the plant a year earlier was very simple in architecture, no bells and whistles, covering a defined area with a multitude of access points into the restricted zones. Among the large number of various sensors, the especially temperamental ones were the through-beam type, which worked on the principle of there being a transmitter and a receiver — if the beam between them is crossed, the alarm trips. People didn't like installing them, because of their strict power requirements and frequent servicing, but, as they say, any whim, as long as the client pays. The volumetric detectors, despite their complexity, with almost a mini-PC on board for preliminary processing of incoming data, bravely withstood the local climate and hardly ever glitched. A volumetric detector "sees" infrared radiation in the monitored zone and reacts to a change in the thermal picture — that is, to the appearance of a person with their body temperature, which is always higher than the surrounding background, no matter what they smear on themselves or what clothes they put on. All these sensors were wired into controllers, small metal boxes that did a little processing on the incoming data, stripping out the noise, and passed it on over the network to the server, where all these events were written to the logs with millisecond precision, with location, time and event type.
We began the check by methodically going around every sensor and resetting it "to zero," as if rebuilding the whole system from scratch — standard commissioning practice when the causes aren't clear. We checked the power-supply logs, where there were no serious spikes or sags over the last couple of weeks, and the uninterruptible power supplies themselves were also in perfect order, the brand-new batteries rated for the standard three years of service before replacement. Out of desperation Max even went to the plant's substation with a couple of "pot-bellied Armenian" bottles to talk to the plant's power engineers — two grey-haired Armenians who'd worked there since Soviet times and probably knew every transformer and every cable in that grid — but there too he was told that everything was running stably, without failures, and that there had been no problems and there would be none.
That week passed surprisingly quietly, without a single failure, without a single burned-out sensor, and I'd already started laughing nervously at the whole situation, blaming the neighbors building the second perimeter, who'd calmed down and decided to behave. Having restarted the system, every day we walked the problem zones, doing routine tuning along the way since we'd gotten out here for unscheduled servicing anyway, checked the sensors' operation, looked at the trip charts from the local whiskered and striped intruders, who had no trouble sneaking up to a couple of scraps of lard left on a threshold by a careless guard while bypassing the super-sensitive detectors. I was just about to call the director and ask him to send someone from the other facilities, because we still hadn't found the cause and the system had stabilized on its own, which looked like a genuinely bad batch of sensors failing one after another. We had no understanding of the true cause of the problem when, on Monday, literally a few hours after we'd come back to the hotel, one of the sensors died right in front of us, at the very moment we were walking past it. What was worse, it was the very sensor we'd replaced on the first day after arriving — brand new, fresh out of the box and from a different batch that had been running for months at other facilities without a hitch — and now it was already dead, just a week after installation. Max looked at me, and in his eyes I saw the same feeling I had at that moment — surprise, irritation and a kind of almost superstitious fear of the inexplicable.
We had to dig back into the event logs with the ferocity of people who'd finally spotted a lead, dumped all the events from the last two months, the time each sensor failed, and matched them against the system logs, where employees' passages through every electronically locked door were recorded. Purely mechanically I sat down to write a script that overlaid the door passages onto the locations of the sensor failures; I had to do a bit of conjuring and process the "smoke break" tables — passages through the side door, which nobody usually bothered to analyze seriously — to pull that data too, not expecting to find anything important there, just to see the correlation between passages and failures. And then, paging through these reports on the operator's screen, I saw a coincidence, a coincidence so clear and so obvious that my first thought was a bug in the scripts, that something had been computed wrong, and I had to recheck it by hand against the paper entry-and-exit ledgers with their signatures. It may sound strange now, but this was the mid-2000s, and paper bureaucracy was very reluctant to surrender its positions to glitchy high technology.
One and the same electronic pass had registered near most of the sensors shortly before they failed — the same face, the same person showing up in the sensors' coverage area, and some time later, anywhere from a few hours to two days, the sensors would die. Not all of them, not every time, but in seven cases out of ten the correlation was so obvious and statistically significant that it couldn't be explained by simple coincidence. In the evening Max and I discussed the results, and at first he looked skeptical, invoking the dreaded word that starts with S a few times, and then sat down to recheck my calculations.
Bleary from lack of sleep, Max was as gloomy as an autumn Piter sky — "We need to see who this is" was his first phrase in the morning, and we went to the head of security with our printout, our map of passages and failures, and he, just as emotionless as ever, sent us off to the submarine graveyard: "Captain First Rank S., he's here temporarily, overseeing the decommissioning process."
← All notes