Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ubuntu repositories broken #6381

Closed
stefanjarina opened this issue Jun 16, 2021 · 95 comments
Closed

Ubuntu repositories broken #6381

stefanjarina opened this issue Jun 16, 2021 · 95 comments
Assignees

Comments

@stefanjarina
Copy link

stefanjarina commented Jun 16, 2021

Hello,

Ubuntu repositories seems to be broken, specifically I am getting '404 not found' for any .deb

from apt update

Err:1 https://packages.microsoft.com/ubuntu/20.04/prod focal/main amd64 dotnet-host amd64 5.0.7-1
  404  Not Found [IP: 104.214.230.139 443]
Err:2 https://packages.microsoft.com/ubuntu/20.04/prod focal/main amd64 dotnet-hostfxr-3.1 amd64 3.1.16-1
  404  Not Found [IP: 104.214.230.139 443]
Err:3 https://packages.microsoft.com/ubuntu/20.04/prod focal/main amd64 dotnet-runtime-deps-3.1 amd64 3.1.16-1
  404  Not Found [IP: 104.214.230.139 443]
Err:4 https://packages.microsoft.com/ubuntu/20.04/prod focal/main amd64 dotnet-runtime-3.1 amd64 3.1.16-1
  404  Not Found [IP: 104.214.230.139 443]
Err:5 https://packages.microsoft.com/ubuntu/20.04/prod focal/main amd64 aspnetcore-runtime-3.1 amd64 3.1.16-1
  404  Not Found [IP: 104.214.230.139 443]
Err:6 https://packages.microsoft.com/ubuntu/20.04/prod focal/main amd64 dotnet-apphost-pack-3.1 amd64 3.1.16-1
  404  Not Found [IP: 104.214.230.139 443]
Err:7 https://packages.microsoft.com/ubuntu/20.04/prod focal/main amd64 dotnet-sdk-3.1 amd64 3.1.410-1
  404  Not Found [IP: 104.214.230.139 443]
E: Failed to fetch https://packages.microsoft.com/ubuntu/20.04/prod/pool/main/d/dotnet-host/dotnet-host-5.0.7-x64.deb  404  Not Found [IP: 104.214.230.139 443]
E: Failed to fetch https://packages.microsoft.com/ubuntu/20.04/prod/pool/main/d/dotnet-hostfxr-3.1/dotnet-hostfxr-3.1.16-x64.deb  404  Not Found [IP: 104.214.230.139 443]
E: Failed to fetch https://packages.microsoft.com/ubuntu/20.04/prod/pool/main/d/dotnet-runtime-deps-3.1/dotnet-runtime-deps-3.1.16-x64.deb  404  Not Found [IP: 104.214.230.139 443]
E: Failed to fetch https://packages.microsoft.com/ubuntu/20.04/prod/pool/main/d/dotnet-runtime-3.1/dotnet-runtime-3.1.16-x64.deb  404  Not Found [IP: 104.214.230.139 443]
E: Failed to fetch https://packages.microsoft.com/ubuntu/20.04/prod/pool/main/a/aspnetcore-runtime-3.1/aspnetcore-runtime-3.1.16-x64.deb  404  Not Found [IP: 104.214.230.139 443]
E: Failed to fetch https://packages.microsoft.com/ubuntu/20.04/prod/pool/main/d/dotnet-apphost-pack-3.1/dotnet-apphost-pack-3.1.16-x64.deb  404  Not Found [IP: 104.214.230.139 443]
E: Failed to fetch https://packages.microsoft.com/ubuntu/20.04/prod/pool/main/d/dotnet-sdk-3.1/dotnet-sdk-3.1.410-x64.deb  404  Not Found [IP: 104.214.230.139 443]

Tried to re-download the repository package, however it seems to be simply not there:

~> wget https://packages.microsoft.com/config/ubuntu/20.04/packages-microsoft-prod.deb -O packages-microsoft-prod.deb
--2021-06-16 22:40:24--  https://packages.microsoft.com/config/ubuntu/20.04/packages-microsoft-prod.deb
Resolving packages.microsoft.com (packages.microsoft.com)... 13.80.99.124
Connecting to packages.microsoft.com (packages.microsoft.com)|13.80.99.124|:443... connected.
HTTP request sent, awaiting response... 404 Not Found
2021-06-16 22:40:24 ERROR 404: Not Found.

This issue seems to be there only for .deb files though, as others can be downloaded without issues, e.g.

~> wget https://packages.microsoft.com/config/ubuntu/20.04/mssql-server-2019.list -O mssql-server-2019.list
--2021-06-16 22:52:28--  https://packages.microsoft.com/config/ubuntu/20.04/mssql-server-2019.list
Resolving packages.microsoft.com (packages.microsoft.com)... 40.114.136.21
Connecting to packages.microsoft.com (packages.microsoft.com)|40.114.136.21|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 101 [application/octet-stream]
Saving to: ‘mssql-server-2019.list’

mssql-server-2019.list        100%[=================================================>]     101  --.-KB/s    in 0s

2021-06-16 22:52:28 (29.2 MB/s) - ‘mssql-server-2019.list’ saved [101/101]

~> cat mssql-server-2019.list
deb [arch=amd64,armhf,arm64] https://packages.microsoft.com/ubuntu/20.04/mssql-server-2019 focal main

This is true for ALL Ubuntu versions

@bquantump
Copy link

Broken here too

wget https://packages.microsoft.com/config/ubuntu/20.04/packages-microsoft-prod.deb -O packages-microsoft-prod.deb
--2021-06-16 20:58:37-- https://packages.microsoft.com/config/ubuntu/20.04/packages-microsoft-prod.deb
Resolving packages.microsoft.com (packages.microsoft.com)... 13.66.21.183
Connecting to packages.microsoft.com (packages.microsoft.com)|13.66.21.183|:443... connected.
HTTP request sent, awaiting response... 404 Not Found


@charliemenke
Copy link

Seeing similar issue for:
Failed to fetch https://packages.microsoft.com/debian/10/prod/pool/main/m/msodbcsql17/msodbcsql17_17.7.2.1-1_amd64.deb 404 Not Found [IP: 13.84.47.71 443]

@akiehl2000
Copy link

Same here:
Screen Shot 2021-06-16 at 4 19 58 PM

@michaelgsharp
Copy link
Member

@rbhanda are you able to help here? This seems similar to the situation we saw the other day #6371

@rbhanda
Copy link
Contributor

rbhanda commented Jun 16, 2021

Thanks for letting me @michaelgsharp . I am following up on this with our infrastructure team.

@rbhanda
Copy link
Contributor

rbhanda commented Jun 16, 2021

Update: Our infra team is working on this. There is an issue with some of the mirrors on packages.microsoft.com so as per them, the current ETA to resolve this issue is in next two hours or so.

@SudhirNune
Copy link

@rbhanda - can you please provide me an update ?

@rbhanda
Copy link
Contributor

rbhanda commented Jun 17, 2021

Update: our infra team is still working to resolve this issue. They ran into some space issues but this issue should be resolved quickly. Unfortunately, I do not have an ETA yet.

@kitsuyui
Copy link

I also understand this is little-bit off-topic for this project.

But let me just say this; I think the problem is not only package repository was broken. Another side of the problem is about communication.

It might be there some progresses on some task tracking system ore some collaboration tools at Microsoft. But we can't see it.

Following issues were opened and some issues were closed so quickly.
It doesn't show somewhere to go. It doesn't show how long we must wait.
So issues are opened repeatedly here and there. This is spreading between projects outside of Microsoft.

@Niek
Copy link

Niek commented Jun 17, 2021

FYI the Edge repo is broken as well, the whole pool directory is gone: https://packages.microsoft.com/repos/edge/pool/

@phknot
Copy link

phknot commented Jun 17, 2021

FYI the Edge repo is broken as well, the whole pool directory is gone: https://packages.microsoft.com/repos/edge/pool/

Same problem for the Teams repo

@gdams
Copy link
Member

gdams commented Jun 17, 2021

We've also had the bug raised against our Microsoft build of OpenJDK repo microsoft/openjdk#74

@r3volver
Copy link

I can confirm this issue. 404 in every debian or ubuntu .deb I search.

In the meantime, is there a mirror where we could find the packages?

Get:1 http://deb.debian.org/debian buster/main amd64 multiarch-support amd64 2.28-10 [215 kB] Ign:2 https://packages.microsoft.com/debian/10/prod buster/main amd64 odbcinst amd64 2.3.7 Ign:3 https://packages.microsoft.com/debian/10/prod buster/main amd64 unixodbc-dev amd64 2.3.7 Ign:4 https://packages.microsoft.com/debian/10/prod buster/main amd64 odbcinst1debian2 amd64 2.3.7 Ign:5 https://packages.microsoft.com/debian/10/prod buster/main amd64 libodbc1 amd64 2.3.7 Ign:6 https://packages.microsoft.com/debian/10/prod buster/main amd64 unixodbc amd64 2.3.7 Err:7 https://packages.microsoft.com/debian/10/prod buster/main amd64 msodbcsql17 amd64 17.7.2.1-1 404 Not Found [IP: 40.114.136.21 443] Err:2 https://packages.microsoft.com/debian/10/prod buster/main amd64 odbcinst amd64 2.3.7 404 Not Found [IP: 40.114.136.21 443] Err:3 https://packages.microsoft.com/debian/10/prod buster/main amd64 unixodbc-dev amd64 2.3.7 404 Not Found [IP: 40.114.136.21 443] Err:4 https://packages.microsoft.com/debian/10/prod buster/main amd64 odbcinst1debian2 amd64 2.3.7 404 Not Found [IP: 40.114.136.21 443] Err:5 https://packages.microsoft.com/debian/10/prod buster/main amd64 libodbc1 amd64 2.3.7 404 Not Found [IP: 40.114.136.21 443] Err:6 https://packages.microsoft.com/debian/10/prod buster/main amd64 unixodbc amd64 2.3.7 404 Not Found [IP: 40.114.136.21 443] Fetched 215 kB in 1s (373 kB/s) Unable to correct missing packages. E: Failed to fetch https://packages.microsoft.com/ubuntu/16.04/prod/pool/main/u/unixodbc/odbcinst_2.3.7_amd64.deb 404 Not Found [IP: 40.114.136.21 443] E: Failed to fetch https://packages.microsoft.com/ubuntu/16.04/prod/pool/main/u/unixodbc/unixodbc-dev_2.3.7_amd64.deb 404 Not Found [IP: 40.114.136.21 443] E: Failed to fetch https://packages.microsoft.com/ubuntu/16.04/prod/pool/main/u/unixodbc/odbcinst1debian2_2.3.7_amd64.deb 404 Not Found [IP: 40.114.136.21 443] E: Failed to fetch https://packages.microsoft.com/ubuntu/16.04/prod/pool/main/u/unixodbc/libodbc1_2.3.7_amd64.deb 404 Not Found [IP: 40.114.136.21 443] E: Failed to fetch https://packages.microsoft.com/ubuntu/16.04/prod/pool/main/u/unixodbc/unixodbc_2.3.7_amd64.deb 404 Not Found [IP: 40.114.136.21 443] E: Failed to fetch https://packages.microsoft.com/debian/10/prod/pool/main/m/msodbcsql17/msodbcsql17_17.7.2.1-1_amd64.deb 404 Not Found [IP: 40.114.136.21 443] E: Aborting install.

@hussaynv
Copy link

Some people suggesting its an issue isolated to Europe, other regions as able to build
https://docs.microsoft.com/en-us/answers/questions/439491/pyodbc-install-error-in-azure-databricks-file-not.html

@sciyako
Copy link

sciyako commented Jun 17, 2021

14 hours! this is blocking us in the production environment right now. as we can't build new machines/vms in Azure.

@GiaNTizmO
Copy link

same too with Debain10 and Ubuntu 20.04LTS

@gmemstr
Copy link

gmemstr commented Jun 17, 2021

Might be worth holding off on the "me too!" or "+1" comments unless there's some new information to make the issue easier to follow.

@Albonycal
Copy link

Space issues?
LOL

@tomas-forsman
Copy link

tomas-forsman commented Jun 17, 2021

Ask Bill to buy some disks from Amazon ;-)

@CapitaineJSparrow
Copy link

Using a VPN and ipconfig /flushdns i was able to download azure cli since europe still down

@abhishek179
Copy link

apt-get fails to get the package from this repo.
Get:1 https://packages.microsoft.com/ubuntu/18.04/prod bionic/main amd64 blobfuse amd64 1.3.7 [7,195 kB].

Is this a known issue that is getting worked upon?

@thedimas3007
Copy link

thedimas3007 commented Jun 18, 2021

Now it works but slowly

@abhishek179
Copy link

yeah it is still slow(intermittently) but working..

@aashu-kumari-by
Copy link

Yeah ours is running only from last 2 hours not sure when it will complete :( @rbhanda Please let me know when we can expect this to be completely working.

@dimanchezzz
Copy link

This solution helped for me
snap install dotnet-sdk --classic

@m17kea
Copy link

m17kea commented Jun 18, 2021

I don't think it is appropriate for Microsoft to proclaim to the world in their own health dashboards and event trackers that all is well, when these services are still clearly degraded.

@aashu-kumari-by
Copy link

Yeah ours is running only from last 2 hours not sure when it will complete :( @rbhanda Please let me know when we can expect this to be completely working.

Now i can confirm its running fine, but yeah we need to monitor it for few more times.
Thanks

@Mlocik97
Copy link

I'm just downloading code update and it's ~500kbps so slow.....

@Uter1007
Copy link

60kb/sec for the azure-cli :( Still hope that microsoft will fix their slow connection for ubuntu repositories.

@ravindrabhartiya
Copy link

Confirming the incident resolution at 10:17PM PST last night. We have restored the service in all regions, been monitoring the results since resolution and got confirmation from multiple customers that they no longer observe the issue. Thank you for reporting this issue.

@tksrc
Copy link

tksrc commented Jun 20, 2021

Hey guys,

I am still having issues interacting with the repo...

#10 6.790 deb [arch=amd64] https://packages.microsoft.com/repos/azure-cli/ hirsute main
#10 6.961 Hit:1 http://archive.ubuntu.com/ubuntu hirsute InRelease
#10 7.009 Hit:2 http://archive.ubuntu.com/ubuntu hirsute-updates InRelease
#10 7.070 Hit:3 http://archive.ubuntu.com/ubuntu hirsute-backports InRelease
#10 7.096 Hit:4 http://security.ubuntu.com/ubuntu hirsute-security InRelease
#10 7.156 Ign:5 https://packages.microsoft.com/repos/azure-cli hirsute InRelease
#10 7.215 Err:6 https://packages.microsoft.com/repos/azure-cli hirsute Release
#10 7.215   404  Not Found [IP: 13.81.215.193 443]
#10 7.306 Reading package lists...
#10 7.837 E: The repository 'https://packages.microsoft.com/repos/azure-cli hirsute Release' does not have a Release file.
------
executor failed running [/bin/sh -c DEBIAN_FRONTEND="noninteractive"  apt-get install -yq ca-certificates curl apt-transport-https lsb-release gnupg         && curl -sL https://packages.microsoft.com/keys/microsoft.asc | gpg --dearmor | tee /etc/apt/trusted.gpg.d/microsoft.gpg > /dev/null         && AZ_REPO=$(lsb_release -cs)         && echo "deb [arch=amd64] https://packages.microsoft.com/repos/azure-cli/ $AZ_REPO main" | tee /etc/apt/sources.list.d/azure-cli.list         && apt-get update         && apt-get install -yq azure-cli         && apt-get clean]: exit code: 100

@tksrc
Copy link

tksrc commented Jun 20, 2021

My bad... I was using ubuntu 21.04 (hirsute). I noticed that version is not yet supported (doesn't exist) in the repo: https://packages.microsoft.com/repos/azure-cli/dists/

Switched back to 20.04 (focal) and it worked.

Thanks

@michaelgsharp
Copy link
Member

Closing since the issue has been resolved.

@MinzigesTV
Copy link

pi@raspberrypi:~ $ sudo apt-get install -y dotnet-runtime-5.0
Reading package lists... Done
Building dependency tree
Reading state information... Done
E: Unable to locate package dotnet-runtime-5.0
E: Couldn't find any package by glob 'dotnet-runtime-5.0'
E: Couldn't find any package by regex 'dotnet-runtime-5.0'

It somehow does not work...

@janekska
Copy link

pi@raspberrypi:~ $ sudo apt-get install -y dotnet-runtime-5.0
Reading package lists... Done
Building dependency tree
Reading state information... Done
E: Unable to locate package dotnet-runtime-5.0
E: Couldn't find any package by glob 'dotnet-runtime-5.0'
E: Couldn't find any package by regex 'dotnet-runtime-5.0'

It somehow does not work...

+1
still the same problem for me too

Used these receipes:
https://docs.microsoft.com/en-us/dotnet/core/install/linux-ubuntu#2004-
and
https://docs.microsoft.com/en-us/dotnet/core/install/linux-ubuntu#apt-troubleshooting

@xTEddie
Copy link

xTEddie commented Aug 10, 2021

I am having trouble install az package

image

@ijnek
Copy link

ijnek commented Aug 10, 2021

@xTEddie That issue is probably related to the packages.microsoft.com outage that has happened today. (Although when you posted, the issue was not up yet). See #6560.

@noxish
Copy link

noxish commented Aug 30, 2021

cannot install dotnet5 on ubuntu, is this still a problem?

@janekska
Copy link

janekska commented Aug 30, 2021 via email

@dralley
Copy link

dralley commented Sep 16, 2021

@rbhanda @leecow What is the correct place to file issues with repositories located at https://packages.microsoft.com/?

Specifically, I have noticed some issues with https://packages.microsoft.com/rhel/8/prod/, but I have not been able to find the correct venue to report them.

@rbhanda
Copy link
Contributor

rbhanda commented Sep 16, 2021

Hello @dralley as of now we do not have a repository for https://packages.microsoft.com/ on GitHub. Until we have one, if you are facing any issues, I recommend to create them on https://github.com/dotnet/core/issues

@hrai
Copy link

hrai commented Nov 10, 2021

I'm on Ubuntu Impish 20.10.

The command given at https://docs.microsoft.com/en-us/dotnet/core/install/linux-ubuntu#2110- refers to 20.10

image

But when I go to https://packages.microsoft.com/config/ubuntu/, there's no 20.10

image

I had to change the command to below:

wget https://packages.microsoft.com/config/ubuntu/21.04/packages-microsoft-prod.deb -O packages-microsoft-prod.deb
sudo dpkg -i packages-microsoft-prod.deb
rm packages-microsoft-prod.deb

Then it worked like a charm.

Perhaps before pushing out documentation, it would be good to test and verify the commands in them.

@mairaw
Copy link
Contributor

mairaw commented Nov 10, 2021

@hrai it would be great if you open feedback for the doc in question. There's a feedback button on the bottom of the page. /cc @adegeo

@adegeo
Copy link

adegeo commented Nov 16, 2021

To close the loop, this was updated in the doc: using the 21.04 repository is the current guidance.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests