Rude, unfriendly, incompetent and unacceptable. Poor service is written large at Microsoft. Microsoft's philosophy: Take it or leave it. There is no support.
This article was first published at ITK SECURITY.
Last update 1 year ago by Patrick RuppeltReading time: 32 minutes
Often enough there are enquiries from customers that even we cannot solve immediately. Because especially when it comes to Cloud products you always have to rely on the manufacturer and that can turn out to be a real challenge.
Without going into the technical details of this case, which would require some background knowledge about mail servers and DNS servers, I will try to briefly explain the background of the case very roughly and as far as possible without using technical terms.
The problem from the customer's point of view is simply formulated: when sending messages to his parent company, he receives an error message and the message is not delivered. Microsoft support didn't give a damn for four weeks.
It is not important for this article to understand the technical facts exactly. I am only concerned with showing that the request If the relevant expertise is available - which one should be able to expect from a mail server administrator at Microsoft - total trivial is.
The facts
A customer has a Microsoft Office 365 mail server at Microsoft in the Cloud. He sends a message to another Microsoft mail server and receives an undeliverable message with error details and recommendations for action:
In detail we see here:
- Sender has a .onmicrosoft.com domain, so it is clearly in the cloud at Microsoft. This means that we have no access to some elementary functions and Microsoft is "the email admin".
- The error is: "Misconfigured PTR Record", i.e. an incorrect so-called PTR entry on the server (more on what this is in a moment).
- Suggested action: you should forward this error message to your email admin, in this case to Microsoft.
This PTR record, which is criticised in the error message, serves in this case, simply formulated, to protect against spam and other harmful content.
The idea is that a mail server always has a so-called IP address. With this IP address, the server reports to the recipient server when sending messages. This is technically mandatory.
Example: Let's take our own mail server. The server has the IP 94.230.50.135. So when we send a mail to someone, the recipient's server knows "Attention, there is a message coming from 94.230.50.135."
The recipient server now only has to find out whether the IP address really matches the sender.
And this can be done with a super simple (Linux) command. One enters "dig -x" followed by the IP address of the mail server and immediately receives as a reply the indication of which mail server belongs to this IP address.
In our example, it would look like this:
Explanation of the example:
- With the query "dig -x 94.230.50.135" I ask which mail server belongs to it.
- The answer "mx0.rzmuc.net" clearly tells me that our own rzmuc mail server belongs to it. So if someone receives a mail from 94.230.50.135 that does not come from mx0.rzmuc.net, then it is most likely spam, a misconfigured mail server or some other attempt at fraud. Simple.
Back to the case of our client. The check in this case shows that there is a misconfiguration at Microsoft in the cloud. There is no valid PTR record.
Since the server is located in the cloud at Microsoft, we have no choice but to report a support case to Microsoft.
Those who have understood a little about what this is all about can probably guess that fixing this error will take less than two minutes:
- Enter "dig -x [IP]" and find out that the entry is really missing, just as the original Microsoft error message says.
- Set the PTR entry on the responsible DNS server (20 seconds of work). Done. Easy. Very easy indeeed.
Enjoy reading about how Microsoft deals with such a support case.
∞
Week 1: And they did not know what they were doing
21. jan 2020 / 13:07
- Reporting a new support case at Microsoft through us
- Provision of detailed error description including screenshots, diagnostic information
- Support consent form signed
21. jan 2020 / 13:08
- Acknowledgement of receipt through Microsoft Support
22. jan 2020 / 10:38 h
- Microsoft reveals to us that whose support requires a conference call between myself, the Microsoft support team and our distributor. (who sells us the licences). Uh, no, it's clear, a telco. Why?
- Quote: "Please let us know for this Two available dates stating the desired call number and time."
- The ticket is handed over "to the specialists", I get contact person number 2.
22. jan 2020 / 11:06 h
- I tell Microsoft support that we do not need a conference call to discuss clear errors (in retrospect, this was a mistake, one should never hold the same against incompetent support staff).
- Once again, I request that the Microsoft technical team fix this clear error.
23. jan 2020 / 08:54 h
- The case was too complicated for contact persons 1 and 2, now I have landed with number 3 with the professionals.
- Supposedly they tried to reach me. Microsoft always says that, even if they never try. Our hotline is available 24/7, 365 days a year. Strangely enough the first excuse given by Microsoft Support in practically every support case so far has always been that we were never reachable. Every now and then we see ping calls (1x ring and hang up immediately) from Microsoft on Saturday evening at 22:00. If that is enough of an excuse for Microsoft, so be it.
- Whatever, I am asked to carry out technical diagnoses and answer a catalogue of questions. With absurd questions, because I have already provided everything that is asked:
> complete NDR - guide will be uploaded
Source: Microsoft Support response from Thu, 23 Jan 2020 07:54:49 +0000
> Could you give me more information why the recipient rejects the mails we send?
> Is there an outgoing IP that it is using that does not have a PTR entry?
- Seriously, I wonder why my initial request contained the most detailed diagnostic information, of course I also submitted the NDR ("non-delivery report", i.e. the original error message from the customer) etc.. The Stalling we have almost skipped and are apparently right at Wear down arrived.
23. Jan, 2020 / 3:56 pm
- My time is both precious and limited. My response to Microsoft:
Hello Ms XXXX,
Thank you for your message. I really tried to answer the questions bit by bit, but then gave up at some point when I thought to myself, Why should I actually fix a faulty DNS configuration from Microsoft??
I am attaching the error message again here. Maybe Microsoft can find out something from this. If not, I honestly don't care, then I'll just publish an article saying Mircosoft can't get it right and send the link to my customer.
The scenario is very simple: Office 365 customer sends an email to someone else. The other person checks PTR records and discovers that they are missing or incorrect. Microsoft Docs are just as little help to me as Microsoft's categorical requests that I be called on the phone.
By the way, I disagree that Microsoft allegedly tried to reach us by phone. This is a standard phrase that Microsoft support generally always states for EVERY TICKET. But it's simply a lie, our hotline is available 24/7/365 and every call is tracked 😉
To be on the safe side: Phone +49-89-4161450-20 (even if this is superfluous, because I can't contribute anything if the Microsoft technicians don't have a clue about DNS).
Kind regards
Patrick Ruppelt
Source: My message to Microsoft Support from 23 Jan, 2020 / 15:56
24. jan 2020 / 10:28 h
- I am told that my request has been forwarded again. I stop counting.
24. Jan, 2020 / 10:45 a.m.
- I say thank you and wait.
Week 2: Nobody understands me
27 Jan, 2020 / 11:23 h
- Three days later, the manufacturer's support team gets back to me.
- Quote:
Dear Ladies and Gentlemen,
Thank you very much for your feedback!
Unfortunately, from this information I cannot understand what exactly it is. I would just need answers to my questions and to have an original email to follow up on them.
> Could you give me more information why the recipient rejects the mails we send?
> Is there an outgoing IP that it is using that does not have a PTR entry?
> When you receive the NDR, please open the NDR message in Outlook in a new window and then click on File, then Save as and save it in msg. format.
Yours sincerely,
XXXX
Source: Microsoft Support response on 27 Jan, 04:23 EST (emphasis mine).
- The questions seem familiar to me. They are the same as in the previous message, i.e. the ones I already answered with my first question. It continues demoralised.
- And the rest... so when the specialised manufacturer support reveals to me in the ninth correspondence that he has no idea about the technical background.then please put six and escalate to a competent staff member.
27. jan, 2020 / 1:17 pm
- Because we probably won't get anywhere any other way, So once again I answer all the questions.
Hello Mrs XXXXX, hello XXXXX,
I am replying to your questions once again.
"Unfortunately, from this information, I cannot understand what it is exactly."
Answer: If you do not understand error messages from your own Microsoft Office 365 mail servers, please forward my request to a competent technician.
"I would just need answers to my gragen and to have an original email to continue working on these."
Answer: I have already sent you the original email. Please find it on your own. I have better things to do than provide you with the same information over and over again.
"Could you give me more information as to why the recipient rejects the mails we send?"
Answer: This is also stated in the Microsoft error message. Short version: see subject of this ticket.
"Is there an outgoing IP that it is using that does not have a PTR record?"
Answer: The outgoing IP is that of your Microsoft Office 365 server. Clearly you lack the expertise to handle the case. I ask again for escalation to a competent employee.
"If you receive the NDR, then please in Outlook open the NDR message in a new window and then click on File, then Save as and save it as msg. Format."
Answer: I have already provided you with the NDR as a screenshot and as plain text. Creating .msg files is not possible for me because I do not have access to the client's mailbox (and am not legally allowed to).
With kind regards
Patrick Ruppelt
Source: My reply to Microsoft Support dated 27 Jan, 2020 / 13:17 (highlighting of my replies for better readability in this article by me).
28. jan 2020 / 14:49
- Even I didn't expect what happened next. Microsoft considers the case solved and closes the ticket without further inquiry.
Dear Ladies and Gentlemen,
I thank you for the good cooperation.
We will temporarily archive the service request until you send us feedback. If you have any further questions, you can always send us a new service request free of charge. We will be happy to take care of your request further.
(...)
Source: Message from Microsoft on 28 Jan 2020 13:49:35 +0000
28. Jan, 2020 / 5:11 pm
- I ask again for Escalation to the superior.
Hello Ms XXXXX,
I don't feel taken seriously.
I answered all your questions yesterday for the umpteenth time, however misleading and stupid the questions may have been.
Why do you claim that I did not answer?
Please escalate the ticket to your supervisor.
Kind regards
Patrick Ruppelt
30. jan, 2020 / 1:37 pm
- That also worked great. Two days later I get the message that I had not delivered an NDR and therefore the case cannot be processed. It is closed again. Lie.
Thank you for your feedback.
Microsoft will now temporarily close your ticket until a full non-delivery report (NDR) is available, as this appears to be necessary for problem resolution.
We wish you another pleasant day.
Source: Message from Microsoft Support on 30 Jan, 07:37 EST.
30. jan, 2020 / 1:43 pm
- Even my patience as a saint when dealing with manufacturers' enquiries (in contrast to the patience I normally display or the non-patience I often have to be told) is slowly being exhausted.
- I'll answer as follows, to be on the safe side, I'll also send it to my distributor so that he's in the picture (by the way, my answer already contains the NDR 100 times in the course of the e-mail, because I had forwarded it originally and my answers always contain the course of the conversation):
Hello Ms XXXX,
how many times should I send the full NDR?
I have stopped counting how many times I have sent this to you. Here again.
We can play this game for longer, I certainly won't stop answering your nonsensically archived tickets over and over and over again until a solution is finally brought about.
Source: My message to Microsoft Support on 30 Jan, 2020 / 13:43 (emphasis mine).
30 jan, 2020 / 13:54
- I'll call my distributor. The contact persons there are generally the first point of contact for us as Microsoft CSP partners, even if they only forward everything to Microsoft anyway and can no longer act as a mouthpiece.
- Unfortunately, no one there has any idea what I'm talking about either. I remember the phone call well, my internal note in our ticket system looked like this:
- As the IT contact for your customer, you are increasingly asking yourself how you can still support cloud systems at all. When manufacturers and distributors are absolutely incapable of understanding problems at allhow would it ever be possible for the manufacturer to rectify the fault in a timely manner?
30 Jan. 2020 / 13:58
- My distributor tells me that he has forwarded my feedback to Microsoft.
Week 3: The silence in the forest
04. Feb, 2020 / 12:15 pm
- Five days later I still have no answerLet alone a plan to solve the problem. After all, my client cannot send emails to certain recipients, and has been unable to do so for a fortnight.
- So once again I ask myself about the state of affairs:
Hello Ms XXXX,
please let me know an action plan for this case.
If this is not possible, please escalate the ticket to your team leader.
After two weeks of inactivity from support, I finally have to tell my customer by when the problem will be solved.Kind regards
Source: My message to Microsoft from 04 Feb, 2020 / 12:15
Patrick Ruppelt
05. Feb, 2020 / 4:44 pm
- Answer from support: they had forwarded the message to the person in charge. Once again.
07. Feb, 2020 / 3:10 pm
- Also another two days later I still have no answer received. Microsoft is probably overwhelmed with my complicated request.
- I ask for escalation to the supervisor, this really can't go on any longer.
Hello Ms XXXX,
please escalate the ticket to your team leader.
After 17 days of complete inactivity from support and no response other than "we have forwarded it", I finally have to tell my customer by when the problem will be solved.Kind regards
Patrick Ruppelt
Source: My message to support on 07 Feb, 2020 / 15:10
Week 4: First there was nothingness and then the big bang
11. Feb, 2020 / 09:49 Uhr
- I'm completely ignored by now. It feels like a long time ago, before the creation of our universe. Even waiting again for Three more daysThe time that has passed in the country does not change the radio silence on the part of Microsoft.
- I'm asking for an update again, I really have to pull myself together to be polite:
Hello Ms XXXX,
I would like an update on the status.
Has the ticket been escalated? Three weeks without any processing and without answers now...
Kind regards
Patrick Ruppelt
Source: My renewed enquiry to the manufacturer support on 11 Feb, 2020 / 09:49
11. Feb, 2020 / 11:41 Uhr
- Short and sweet answer: " Thank you very much for your message. Since Microsoft has archived your case, we have a new Suport ticket for you under the number #18668XXX opened."
11. Feb, 2020 / 11:57 Uhr
- The point has been reached, I don't like it any more. I can't do it any more. You're killing me, you incompetent people.
- I'm writing to my distributor, after all he collects all the money for the licences and paid support agreements, so he should do something about it:
Hello Ms XXXX,
I am not interested if you or Microsoft permanently archive unresolved tickets without notifying me and then open new ones where the response time starts from zero again and I have to provide all the information again.
I have been waiting for three weeks for processing. XXXX is my CSP partner, so please take care of it!
Please tell me the name of your team leader so that I can discuss the matter with him. Unfortunately, this is not the first time, but a fundamental problem.
XXXX support doesn't know about the products and Microsoft doesn't care.
Kind regards
Patrick Ruppelt
Source: my message to Microsoft from 11 Feb, 2020 / 11:57
12. Feb, 2020 / 10:22 Uhr
- Missed call from my distributor. But hey, he was talking to colleagues - our hotline is always busy even when I'm at the customer's - he sends me an email.
12. Feb, 2020 / 1:12 pm
- Now, once again, new staff have taken over and, who would have thought it, we are starting again with Adam and Eve. Yes, We need a phone call between all parties involved, the CSP distributor, myself and the technicians from Mircosoft.:
We tried to contact you today, unfortunately without success.
Would it be possible to arrange another meeting with XXXX from Microsoft? Source: Answer from Support, 12 Feb, 07:12 EST (I'll put the time zones in the original format here from time to time, because it's interesting to see where the support people are located all over the world).
12. Feb, 2020 / 2:06 pm
- My answer is not long in coming:
Hello Mr XXXX,
everything has been bought through several times with Microsoft support. Also by telephone.
Then Microsoft simply closed the ticket without processing it, despite many queries from me.
And now I'm supposed to invest time again to explain everything all over again? Certainly not.
You and Microsoft have ALL the information to handle the case independently.
Please inform your team leader about the incident and provide a solution immediately.
With kind regards
Patrick Ruppelt
Source: My message to manufacturer support on 12 Feb, 2020 / 14:06
Week 5: Microsoft demands lawbreaking
17. Feb, 2020 / 10:13 Uhr
- The manufacturer requests a callback for a telephone conference.
17. Feb, 2020 / 10:29 Uhr
- These games from Microsoft are getting us nowhere. Short and sweet:
Hello Mr XXXX, hello Ms XXXX,
please let me know the contact details of your supervisor. I am still waiting.
With kind regards
Patrick Ruppelt
Source: My message to the manufacturer support on 17 Feb, 2020 / 10:29
17. Feb, 2020 / 5:12 pm
- What is happening now is really crass. There is no other way to put it:
- Microsoft informs me that required to process my request be that I draw up a complete record of all messages that the client has ever sent in the past three months. Including all recipients and message subject lines. This is unnecessary, extremely costly, technically totally absurd, and Legally absolutely inadmissible.
- Also, may I please supply the NDR. Microsoft allegedly never received an error message from me.
- So we are definitely at the Attrition and lies arrived in its pure form.
18. Feb, 2020 / 08:29 Uhr
- It doesn't matter now, I stand firm:
Dear Ladies and Gentlemen,
please let me have the contact details of your supervisor.
With kind regards
Patrick Ruppelt
Source: My reply to the manufacturer support on 18 Feb, 2020 / 08:29
18. Feb, 2020 / 09:58 Uhr
- Microsoft calls again and stupidly catches me in the office. So the following is not in writing, but only my transcript of the conversation in our ticket system:
- So I am accused of being uncooperative.
- Supposedly they have been waiting for information for four weeks, but unfortunately cannot say and also not be able to prove when they would have been requested.
- When I insist on telling me when Microsoft would have requested this information, the Microsoft employee just hangs up and kicks me off the line.
- I start writing this blog article. Unfortunately, this is of little help to the client, who has not been able to send any mails to his parent company for four weeks.
18. Feb, 2020 / 10:27 Uhr
- Microsoft is speaking out again, I may please send the NDR message as well as the extended message tracing. Also, may I please answer two questions (we remember, these are the questions I already answered four weeks ago):
- Could you give me more information why the recipient rejects the mails we send?
- Is there an outgoing IP that it is using that does not have a PTR record?
18. Feb, 2020 / 11:48 a.m.
- It doesn't take me an hour to reply in detail, I only have to copy the answers from my past messages:
Hello Ms XXXX,
attached the annexes and answers to the questions.
I must state once again in advance that I disapprove of Microsoft's approach and am of the opinion that XXXX must also approach Microsoft Support more emphatically here. It is incomprehensible why Microsoft can accuse me of not being cooperative on the one hand, since I have allegedly refused to provide an extended message trace for four weeks, and on the other hand Microsoft cannot tell me when I was supposed to have been requested to do so, my own ticket system saw something of this for the first time yesterday and the XXXX ticket system also saw something of this for the first time yesterday.
This is the only reason why four weeks and about seven hours of support on my side have been "burned" for nothing and nothing again. My life time is too valuable for something like that.
Incidentally, there is not a single e-mail that I have not answered immediately. I resent Microsoft's comments that I would not respond. That is a lie.
If Microsoft already accuses its customers and partners of not cooperating (in order to be able to close tickets without processing them, as has already happened here), then Microsoft should please also prove this and not throw me off the line if Microsoft suddenly finds no correspondence in this regard.
At least an apology from Microsoft would have been appropriate at this point, but instead Microsoft throws me off the line. In my opinion, that is unacceptable.
Having said this, I will answer all your questions and also take the liberty of pointing out when they have already been answered by me:
- "A message tracing"
Response: You will find the message trace as an attachment to this message.
Note 1: Microsoft vehemently claims that it has been waiting for this information for four weeks. As mentioned at the beginning, however, we received the request from Microsoft yesterday for the first time - mind you, after the ticket had already been closed and archived by Microsoft.
Note 2: The download of the message tracing does not work reliably because the Microsoft server almost always returns an HTTP error 503 "Service not available". I have attached a screenshot of the error message from a Windows machine and a Linux machine. After calling the page about 8 times, I finally got the CSV file.
- "complete NDR"
Response: You will find the NDR in msg form type as well as in unicode msg format as an attachment to this message. Furthermore, also as txt format and the original message as eml file. Hopefully you are able to open any of these files.
Note: I have already answered this question explicitly on 30.1.2020. You received all the headers there, admittedly as a txt file and not as an msg file. I was not aware that Microsoft can only process msg headers and that txt headers cannot be read by Microsoft, although even Windows notepad.exe can open a text file. But that doesn't change the basic facts.
- "Could you give me more information why the recipient rejects the mails we send?"
Answer: Yes. "It appears that the recipient's email server at XXXX.de performed a reverse DNS (rDNS) lookup security check to verify that the IP address the message is coming from is associated with the sending domain, and the lookup failed. It appears that the pointer (PTR) record for XXXX.de isn't set up correctly."
Note: I have already answered this question explicitly on 27.1.2020.
- "Is there an outgoing IP that it is using that does not have a PTR record?"
Answer: It is a Microsoft Office 365 server. Only Microsoft knows which outgoing IPs this server uses.
Note: I have already answered this question explicitly on 27.1.2020.
- "Therefore, I would like to ask you to send me a short information about your availability if possible."
Answer: Today Tuesday 18.2., as well as Thursday 20.2. and Friday 21.2. I am available.
Source: My opinion of 18 Feb, 2020 / 11:48
18. Feb, 2020 / 11:50 a.m.
- I call Microsoft Support to make sure that my message has arrived this time. Answer: No, it didn't arrive, but we get that a lot at the moment.
- I am given a link to upload the message and attachments directly one more time.
18. Feb, 2020 / 11:56 Uhr
- I am doing what I was told to do. I upload everything again separately. This time in another ticket system I have never seen before.
18. Feb, 2020 / 12:05 pm
- And again I call support to make sure that they now have access to it. Result: no, Microsoft is having a technical glitch, my replies are missing from the system.
18. Feb, 2020 / 1:01 pm
- The receipt of my messages and attachments is confirmed. Finally, things can only get better.
18. Feb, 2020 / 2:12 pm
- They confirm to me that the case has been referred to the The relevant member of staff in the specialist department is forwarded to was. Again. Nice, but my customer can't buy anything from that.
19. Feb, 2020 / 09:27 o'clock
- The next morning I get the next cheap excuse from Microsoft support, which unfortunately we now hear regularly (and also from many other manufacturers):
- You need a more current Error example, because the error message I provided was already a month old, you can't do anything with it.
- I wonder whose fault that is.
20. Feb, 2020 / 09:57 Uhr
- Mircosoft complains about it, that I would not answer again #Lie:
I have not received any feedback from you regarding further processing.
In this context, I wanted to let you know that I am trying to find or implement a solution to the concern as soon as possible and therefore I would need to get in touch with you. Without your cooperation in this case, no progress will be made in finding a solution.Source: Message from Microsoft on Feb 20, 03:57 EST
- What confuses me, however, is that the message in the Mail history at the end apparently contains an internal note from a "real" technician at Microsoft from that very day. I assume that it is, because firstly it is technically correct for once, secondly it is in English, thirdly it does not address me but is rather general:
I checked the PTR for the Ip present in the NDR xxx.xxx.xxx.xxx and it is correctly configured, so from our side it is ok and the issue has to be checked on the recipient side
The question about PTR is that it should always be configured for our Ip addresses, this is the crucial requirement that should be satisfied, there is/was a second one, the PTR name should not match the EHLO string that our servers provide when connecting, but this second requirement is anachronistic and it is not possible for cloud services.
Source: in the fine print of Microsoft's Feb 19, 03:27 EST response.
- Wow, the first person at Microsoft who understands me. That's the point, "the PTR name should not match the EHLO string that our servers provide when connecting". I've been trying to explain this to Microsoft for almost five weeks.
20. Feb, 2020 / 10:13 Uhr
- So if this morning the first Microsoft employee who knows what a PTR entry is checked this out.... then he will probably have noticed the mistake and corrected it in 20 seconds. At least that's what we should assume, otherwise it would be embarrassing to have to admit a mistake after weeks of discussion.
- I take this as an opportunity to check the PTR Record again.
- Hallelujah! Microsoft Support has actually done it. My customer can finally send emails again after weeks of downtime.
What do we learn from this?
Today I will be very brief about that.
- Processing time: over 1 month
- Impairment at the customer: over 1 month
- Microsoft troubleshooting time: 30 seconds
- Pure work performance by us (without waiting times): approx. 12 hours
If you depend on an application, don't go to the cloud.
