Versione italiana disponibile qui → [IT]
In Part 1, the blackout looked like the perfect culprit. The local database answered, BIGDB seemed alive, the trust existed. And yet the booking system died only when it had to cross the boundary toward headquarters. That is where we start again.
Back to the boundary
When Mateo came back into the office behind reception, he had the face of someone who had just won a small battle but could still see only wreckage around him.
“WiFi?” Sid asked, without taking his eyes off the diagram.
“Access points restarted, captive portal fixed, guests temporarily pacified,” Mateo replied. “I think nobody will threaten negative reviews for at least ten minutes.”
Sid nodded, but he did not smile.
The pen was still there, on the line drawn between the two domains.
Resort on one side. Headquarters on the other.
In the middle, the paradox of that apparently reassuring word: trust.
Could that line really be trusted?
“Before the manager comes back with another emergency,” Sid said, “let us start again from here: from the boundary.”
Mateo sat down.
“What do you want to see?”
“Ask me a stupid question,” Sid said.
Mateo looked at him.
“Has anything changed recently on this side?”
Mateo almost laughed.
“Here? Sid, nobody has touched these servers in so long that if you move the mouse, dust rises from it.”
“That is what I thought,” Sid replied. “And that is exactly what worries me.”
He let a second pass, then pointed to the side of the diagram where Mateo had written headquarters.
“Do we know if anything changed on the other side?”
Mateo did not answer right away. He looked at the phone, then at the laptop, then back at the diagram.
“I can ask,” he said. “But last night, when I spoke to them, they told me everything was green on their side.”
“Everything is green is a dangerous sentence,” Sid replied. “It usually means someone looked at the dashboard, not at the engine.”
“Is there around the clock support?” Sid asked. “If there is, ask whether anything changed.”
Mateo nodded and opened a chat with the central team. The first reply arrived after a few minutes, dry, reassuring and far too simple.
BIGDB is online. No open incident. Monitoring ok.
Sid read the message and looked up at the ceiling as if trying to communicate with someone.
“We did not ask if BIGDB is alive. We asked if anything changed yesterday.”
Under the hood
Mateo was already typing a reply, but Sid stopped him with a gesture. “Wait. Before we start arguing with a dashboard that says everything is green, let us look under the hood ourselves. If the problem is on the boundary line, there must be traces.”
“What kind of traces?” Mateo asked.
“Packets. Conversations. If the booking system tries to talk to headquarters and someone replies with an error, that reply crosses the network. And the dashboard does not show that level of detail.”
He asked Mateo to install a network analyzer directly on the booking application server. Of course, there was none. Nothing had been installed on those machines in years.
“Wireshark?” Mateo asked.
“Wireshark, Network Monitor, whatever you can start without rebooting half the resort. You already have enough trouble. Just capture.”
Mateo started downloading the package, and it took longer than expected.
From the slightly open door came the murmur of the lobby, trolleys, voices and the reception bell cutting through a silence that had become almost absolute in that room.
Then he installed it and finally started the capture.
“Now let us reproduce the problem,” Sid said. “Open the reservations. The ones that come from BIGDB.”
Mateo clicked. Spinner. Waiting. Error. As expected.
“Stop here. We have everything we need,” Sid said.
Mateo stopped the capture. Thousands of lines remained on screen, and he thought, “it looks like the Matrix. How are we supposed to find the right information in this chaos?”
“First rule,” Sid said. “You do not filter to find what you are looking for. First you look at who the machine talked to, then you decide what is useless.”
He did not write any filter. Instead he opened the conversations list, the one that summarizes every peer with its ports, packets and sequences, and sorted it by traffic volume.
At the top, with an embarrassing margin over everything else, there was the Remote Desktop session Mateo was using to connect to that server.
“We are capturing ourselves too,” Sid said, pointing at it. “It always happens. Every movement on the screen becomes traffic and drowns out everything else.”
With two filter lines he removed the remote session, then backup and antivirus traffic, which were generating just as much useless noise. What remained was small, tidy and finally interesting.
“Now we do not look at the content,” Sid said. “We only look at who it talks to and on which ports. The content comes later.”
He scanned the protocol column with the calm of someone who knows that, if there is a pattern, it repeats. And it did. Every time Mateo opened the reservations, the sequence was the same. A name resolution, fast and clean. Then the connection to the BIGDB server opened normally, but there, strangely, everything looked fine. And immediately after that, before the booking system asked for a single row of reservations, there was an exchange toward a port Sid knew by heart.
“Eighty eight,” he said.
Mateo leaned toward the screen. “Does eighty eight point to BIGDB?”
“No. BIGDB has nothing to do with that port, it listens elsewhere. On eighty eight, Domain Controllers answer. That is where Kerberos answers.”
Sid pointed to the sequence from the beginning. “Before reading a reservation, your server asks permission to do it. And it asks there, not at the database.”
Then he pointed to the end of the sequence. After that exchange, the connection to BIGDB closed. It had not managed to ask anything of the actual database.
Mateo stared at the screen. “Wait. So BIGDB had answered.”
“BIGDB answered perfectly. It did exactly what it was instructed to do, for twelve hours, while you accused it of being unreachable. But your server never got as far as asking it anything concrete. It stopped at the entrance, and nobody opened the door.”
Mateo ran his hands over his face. On the table, the notebook with twelve hours of crossed out hypotheses remained, along with the feeling that he had spent the night interrogating the wrong witness.
“And there is one more thing,” Sid added, following the address column with his finger. “On that port, your server talks twice. But not to the same machine.”
Mateo moved closer. He recognised the first address at once: it was the Domain Controller of the resort, the one in the cabinet at the end of the corridor. He had never seen the second one. It had to belong to headquarters.
The door
Sid changed the filter. This time one word was enough, the protocol name. The few hundred lines became a few dozen.
And all those lines had one thing in common. None of them went to BIGDB. In the entire capture, toward the central database, there was not a single Kerberos packet. The negotiation stopped before reaching BIGDB.
He scanned quickly. The first request went to the resort Domain Controller and received a clean answer. But that answer was not the ticket to BIGDB. It was a referral, that is what Kerberos calls it: a ticket that opens no door, it only says “that service is not mine, go and ask them”. That referral was only good enough to present itself beyond the boundary. The second request went to the other address, the one Mateo did not know. And there, almost at the end, one line turned red.
KRB-ERROR
error-code: KDC_ERR_ETYPE_NOSUPP (14)
Sid pointed at it with the pen. “This is not database not reachable. This is someone saying: I cannot speak your language.”
“What language?” Mateo asked.
“Cryptography. Your system asks for a ticket and declares which algorithms it can work with. The other side replies that what it offers is no longer accepted.”
Mateo kept staring at that red line. Twelve hours of generic error had just become a precise message. Relief and annoyance, together.
Sid scrolled a few more lines, then stopped. “And here is the second strange thing. After that no, nothing else happens.”
“What should happen?” Mateo asked.
“Usually, when Kerberos fails, Windows does not simply give up. It tries to fall back to an older method, NTLM. Outdated, vulnerable, often abused, but it has been there for decades for backward compatibility. Here I do not see any trace of it. Not even an attempt. The booking system received the no and that was the end of the game.”
“Before pointing fingers, though, I want to be sure where that no was born.”
Sid opened the event log on the resort Domain Controller and filtered the Kerberos events. Almost nothing. Tickets granted, referrals issued, no errors.
“Everything is clean here, which is what I expected and consistent with the network trace. Your Domain Controller did its job: it gave the booking system the referral to cross the boundary, and from that point on the matter was no longer its responsibility. Everything confirms that the refusal did not start here.”
“Then where?” Mateo asked.
Sid went back to the red packet and pointed to the address that had replied with the error. It was the second one, the address Mateo had not recognised. “That is the one that said no. It is not one of your servers.”
“One of the central Domain Controllers?” Mateo wondered aloud.
“Exactly. The real error does not live in your logs. It lives in theirs.”
“But before we write to them, let us gather the right codes,” Sid said. “If I send them a theory, they will answer with another green dashboard. If I send them codes, they have to go and look.”
The numbers that do not add up
He moved back one line, to the request that had preceded the refusal, and opened the packet detail. The lower pane filled with branches to expand, one inside another, like nesting boxes. Sid went all the way down, where the client declared which encryption types it was willing to use.
There was only one number in that list: 23.
“There it is,” Sid said, and for the first time since entering that office he almost looked amused. “Only one. No alternative. Your system presents itself beyond the boundary saying: I can speak this language, and no other.”
“And what would 23 mean?” Mateo asked.
“RC4,” Sid replied. “It is the number the standards assigned to an encryption algorithm born more than thirty years ago. Here you see it in decimal, 23. In Windows logs you will see it in hexadecimal, 0x17. Same code, two ways of writing it.”
Outside, the wind moved a palm tree. Mateo turned the diagram over to the blank side and wrote the numbers in a column, with the uncertain handwriting of someone who had not slept for a full day.
Then Sid went back to the red line and pointed to the error code. Fourteen.
“And this is the no. Fourteen in decimal, 0xE in hexadecimal. It is the same number that, on the other side, ends up in the field Windows calls Failure Code. The network and the event log are telling the same story, just in two different alphabets.”
“So their event log should have this specific error. How do they find it?” Mateo asked.
Sid leaned back. “That part is quite simple. When a system asks for a ticket for a service, Windows always writes the same event. It has a very specific number: 4769. Unfortunately, I remember it by heart. I have crossed paths with it too many times in my career.”
“And it is written only by the Domain Controller that issues the ticket,” he added. “That is why there is nothing to see here. That refusal was written by a machine outside our control.”
“So now we have three numbers,” Mateo said.
“We have three numbers and one precise check that cannot be ignored,” Sid replied. “And that is a completely different matter from asking if the database is healthy.”
“Write to the central team again. But this time do not ask if BIGDB is fine. We ask something precise,” Sid said, frowning as he put all the pieces back in order.
Mateo wrote under dictation.
On your Domain Controllers, from tonight onward, look for 4769 events in Failure with Failure Code 0xE coming from our domain. Then check the KDCSVC events in the System log. Also tell us which encryption types are configured on the trust object toward us. Confirm whether NTLM is allowed for that service.
Mateo hesitated with his finger over the send key. “And if those events are not there?”
“Then I am wrong, we go back to the packet and start again,” Sid replied. “But the network trace is clear and it does not lie. The events are there, we only need confirmation of who is generating them. It is written in their logs, with their numbers, on their servers.”
Mateo pressed send. In the office, only the ceiling fan remained audible.
The answer came more slowly than the previous ones. And this time it was different.
Confirmed. Hundreds of 4769 failures, code 0xE, all from your domain, all starting tonight. The event details show the mismatch: the client offers only RC4, while the KDC accepts only AES. On the trust object toward you, no AES encryption type is enabled. And I confirm NTLM is not allowed either: we disabled it on that service a few months ago.
“That is why it does not fall back,” Sid said. “They closed its last escape route months ago, and they were right to do it. But it is also why your booking system died instantly instead of silently finding a way around it, as old systems do when nobody is watching.”
Mateo looked at the screen, then at Sid. “This confirms the theory, right?”
“Almost. Now we can ask the right question,” Sid replied, and had him write one last line.
Did you change anything tonight in the Kerberos configuration of the central Domain Controllers?
This time, the answer was not reassuring. It was honest.
Yes. Scheduled maintenance on the central DCs. Monthly patches and Kerberos hardening, with RC4 removal. No expected impact on application services.
Sid read the last line and gave a half smile. “No expected impact,” he repeated. “It is always the last sentence before every disaster.”
Then he took the pen and circled the line between the two domains. “Here is the boundary. Since tonight, headquarters has stopped accepting an old cryptographic language. Your booking system only knows that one. As long as it speaks at home, nobody corrects it. The moment it tries to cross the trust, it is blocked.”
“So at this point the problem is the trust?” Mateo asked.
“The problem is not the trust itself. The problem is that headquarters closed the old algorithm and forgot about you and your trust.”
Misunderstandings
Mateo tapped his temple. “Ok. Then I ask headquarters to allow the old language again, RC4, and we start again. Right?”
Sid slowly shook his head, like someone who had already expected that proposal.
“Of course it would start again. And tomorrow morning someone at headquarters would notice that the old rusty lock they had just thrown away was back on the door. They did not remove RC4 on a whim, Mateo. They removed it because it is weak. Putting it back just to restart a booking system is like turning off the fire alarm because it is too loud.”
“So what do you suggest?” Mateo asked.
“First, we need to understand where the patch belongs,” Sid said, and walked through the sequence on the diagram again. The referral issued on their side, the boundary to cross, the ticket for BIGDB that was never issued. Three steps, and the no always arrived at the second.
“And this is where people usually aim at the wrong target,” Sid continued. “They look at the identity coming from the resort and think: the problem is whoever is knocking. But the resort user, on the other side, is almost a shadow.”
“At headquarters there is only a local representation of your identities. It is called a Foreign Security Principal. It says who you are, but it does not contain your keys. Changing settings there produces no effect.”
A louder wave of voices came from the lobby. A child had started crying because, while running, he had knocked over the line of trolleys. Mateo glanced toward the door for a moment, then went back inside the problem.
“Ok, the calling user has nothing to do with it. Then who does?”
“Two things, in order. The bridge and the destination. The trust, and the service that owns BIGDB. We start with the bridge.”
At that point there was not much left to discover about the trust. There was only a decision to make. The diagnosis was already there, in the capture and in the headquarters logs: the resort referral still travelled with an old key, while on the other side that language had been closed. To be safe, Sid had the headquarters team open the trust configuration too, not to look for a new culprit, but to verify the only thing that mattered: from that side, the relationship toward the resort did not declare AES support. The confirmation arrived in a few minutes. It was not a new lead. It was the stamp on the hypothesis.
“There is the first piece,” Sid said. “The bridge was the part left behind. Not BIGDB, not the local database, not the network. The trust.”
“And the destination?” Mateo asked.
“Right, before we close the loop, let us check the destination too,” Sid said. “Not the database itself. In Kerberos, every service has a technical name by which it is recognised. It is called a Service Principal Name, or SPN. It is the label that tells the Domain Controller which account it must prepare the ticket for. And every SPN is associated with one account.”
Mateo nodded slowly. “So we need to understand which account owns the SPN for BIGDB.”
“Exactly. Not because we are looking for a new culprit. We are removing the last doubt from the table.”
One phone call and a few minutes were enough. Behind the impressive name BIGDB there was a service account, managed by headquarters and, as expected, already included in the hardening path: that account spoke only AES.
Mateo looked at Sid. “So BIGDB is really out of the case.”
“Out of the case,” Sid confirmed. “The problem was not the destination. It was the way the resort tried to reach it.”
Mateo took a breath. “Ok. We understand where the problem is. But how do we fix it without putting the rusty lock back?”
“By teaching the new language to the bridge, instead of forcing headquarters to speak the old one again,” Sid replied.
“The bridge, meaning the trust, must be updated so that it can handle AES too. It is an intervention on the relationship between the two domains, from both sides, and it requires regenerating the keys with which the bridge signs the passage. Be careful though: it is not a simple click, it must be coordinated precisely with the colleagues at headquarters. But it is the best solution.”
Meanwhile the sun had shifted and was entering sideways through the window. Sid moved the laptop to get the reflection out of his eyes.
“And the BIGDB service account?” Mateo asked. “Do we leave that as it is?”
“We do not touch it, it is already fine,” Sid replied. “If anything, we check things on your side. I want to be sure that, once AES is opened on the trust, your Domain Controllers really use it to seal the referral. A simple verification on the source system. The main work is on the trust.”
Mateo glanced at the laptop, with the look of someone who still had one doubt ready. “Wait. But the booking system is ancient. Are you sure it can speak AES?”
“Good question, and exactly the right moment to ask it,” Sid said. “Luckily, the facts are on our side. The booking system never chose RC4. It does not even know what it is. It asks the operating system to authenticate and trusts whatever it receives back. Windows does the dirty work. And your servers, old as they are, can speak AES perfectly well. RC4 is not their choice. It is a setting inherited from a world nobody has touched in a very long time.”
“So no exceptions, no shortcuts,” Mateo said, almost relieved.
Sid paused, the kind of pause that comes before uncomfortable truths. “In an ideal world, no exceptions: you update the trust, verify that the resort can use AES and close RC4 forever. In the real world, though, you have a full lobby, a maintenance window to agree on and headquarters dealing with its own bureaucracy. So there is only one middle way left.”
“Which one?”
“We have to take one forced step backward and ask headquarters to allow RC4 only where it is really needed: on the account that owns the SPN for BIGDB, and only for the time required to bring the service back. That gives us the time needed to work on the trust. And yes, I know, an exception like this is ugly, but it is limited. Above all, it must be written down: who asked for it, why, when it expires and what the plan is to remove it. An exception without an expiry date is not a solution. It is another piece of legacy you are creating right now, with your own hands, while you think you are fixing one.”
Mateo looked at him for a second, then slowly nodded. He had just understood why a temporary solution can be more dangerous than the problem it claims to solve.
To restart the resort, headquarters agreed to apply a temporary exception on the account associated with BIGDB’s SPN. In parallel, they opened the real work: updating the trust, verifying the behaviour of the resort Domain Controllers and planning the final removal of that old cryptographic language.
Does the bill add up?
Mateo went back to the booking system, took a breath like someone about to defuse something, and opened the reservations again.
Spinner. Waiting. And then, for the first time in twelve hours, the room list began to fill again, row after row, like the tide that had started rising again outside the resort.
“Holy Token,” Mateo murmured, without noticing.
Sid smiled. “I see it is contagious.”
A little later the office door opened again, but this time slowly. The manager came in with his shirt still marked by sweat, but with the face of someone who had just received good news. The WiFi was back, reservations were opening, and the families in the lobby had started moving toward their rooms.
“Mr Historia, I do not know how to thank you,” he said, with the air of someone who had just come out of a nightmare. “The suite is yours for the entire stay, of course. And if you like, we can discuss a discount for a future stay. In fact, for anything you need during your stay, call me directly.”
Sid stood up, adjusted his linen jacket and remained silent for a moment, like someone deciding whether to say the comfortable thing or the right one. Then he chose the second, because the first had never come naturally to him.
“I will gladly accept the suite, I will not pretend I did not miss the sea view,” he said. “But allow me one thing, and this is not a criticism of Mateo. Quite the opposite. Today you were not saved by the luck of meeting me in the lobby. You were saved by the fact that this guy spent twelve hours trying to keep something standing that, honestly, should not have been standing on its own.”
The manager opened his mouth to say something, but Sid continued, with the tone of someone who was not scolding, only stating a fact.
“You did not have a simple outage. You had a bill that arrived late. For years someone at headquarters modernized, decommissioned and secured systems. And for years this piece of infrastructure stayed still, in a corner, speaking a language the rest of the world was slowly leaving behind. Everything worked as long as nobody on the other side changed the rules. Tonight they changed them, and the bill arrived all at once. The guests in the lobby were there to witness it.”
“But now everything works again,” the manager tried to say.
“It works again because today you were lucky,” Sid replied, without harshness, but without softening it. “Luck, however, is not a budget line. An infrastructure does not stay alive thanks to a panoramic suite and the hope that the right consultant happens to walk through the lobby at the right time. It stays alive with something far less charming: time, budget, maintenance, people paid to look after these systems before they go into crisis. You need an inventory that knows this resort exists. A plan to bring these systems into the present, one piece at a time, before the next blackout decides it for you.”
He gestured toward the badly drawn diagram, still there on the table, with its pen line separating two worlds.
“That temporary exception we activated has an expiration date. Use it as an alarm clock, do not let it be forgotten. Because the next person who finds this system in this state may not be me on holiday. It may be someone who wants to do serious damage and sends you a ransom note.”
The manager remained silent. For the first time since entering, he did not have a reception smile ready on his face. He looked like someone who had just understood that the real problem had not been solved by Sid. That one could only be solved by him, with a signature on a budget, not on a courtesy voucher.
“I will think about it,” he said at last. And for once it did not sound like a polite formula.
Sid picked up his jacket and headed for the door. Mateo followed him with his eyes, then said, almost shyly: “Thank you. Really.”
“Thank yourself,” Sid replied from the threshold. “I only read a packet. You held an entire resort together with duct tape and stubbornness. But duct tape, as you have understood, is not enough and sooner or later it runs out. Get yourself the right tools before the next blackout asks for them.”
Sid headed toward the sea. Behind him, the resort had started breathing again. The problem was solved. The immediate one, at least.
The lesson, as always
Sid’s story is fictional, but the resort in this story exists in a thousand real variations. It is almost never a system that breaks by itself. It is a system nobody was looking at anymore, kept alive by habit, until one day it meets a world that has moved on without warning it.
Legacy, the real kind, is rarely a technical problem. It is a problem of memory and priorities. That resort was not simply frozen in time. It had slipped out of the organisation’s shared memory, into that grey area where systems keep working precisely because nobody touches them, until the rest of the world changes the locks.