prometheus重启探针报错opening storage failed: lock DB directory: resource temporarily unavailable

prometheus修改配置文件后需要重新加载，重启后prometheus的探针一直在重启，删除重新创建会导致数据丢失，所以还是需要解决这个问题，主要是因为prometheus运行过程中使用的是nobody。报错内容如下：

level=info ts=2022-02-25T03:33:43.791Z caller=main.go:308 msg="No time or size retention was set so using the default time retention" duration=15d

 level=info ts=2022-02-25T03:33:43.791Z caller=main.go:343 msg="Starting Prometheus" version="(version=2.20.0, branch=HEAD, revision=e5a06b483527d4fe0704b8fa3a2b475b661c526f)"

 level=info ts=2022-02-25T03:33:43.791Z caller=main.go:344 build_context="(go=go1.14.6, user=root@ac954b6d5c6e, date=20200722-18:51:45)"

 level=info ts=2022-02-25T03:33:43.791Z caller=main.go:345 host_details="(Linux 3.10.0-1160.15.2.el7.x86_64 #1 SMP Wed Feb 3 15:06:38 UTC 2021 x86_64 prometheus-6d9746f9f5-d5t8g (none))"

 level=info ts=2022-02-25T03:33:43.791Z caller=main.go:346 fd_limits="(soft=1048576, hard=1048576)"

 level=info ts=2022-02-25T03:33:43.791Z caller=main.go:347 vm_limits="(soft=unlimited, hard=unlimited)"

 level=info ts=2022-02-25T03:33:43.803Z caller=main.go:684 msg="Starting TSDB ..."

 level=info ts=2022-02-25T03:33:43.803Z caller=web.go:524 component=web msg="Start listening for connections" address=0.0.0.0:9090

 level=info ts=2022-02-25T03:33:43.811Z caller=repair.go:59 component=tsdb msg="Found healthy block" mint=1645007760362 maxt=1645012800000 ulid=01FW1BYE3JYBF7HJYZJW8R4VM2

 level=info ts=2022-02-25T03:33:43.813Z caller=repair.go:59 component=tsdb msg="Found healthy block" mint=1645012800000 maxt=1645077600000 ulid=01FW3EK6JDC8A2E328C9763MEG

 level=info ts=2022-02-25T03:33:43.815Z caller=repair.go:59 component=tsdb msg="Found healthy block" mint=1645077600000 maxt=1645142400000 ulid=01FW5CCQX2E3C5KD4YRAFR5FEM

 level=info ts=2022-02-25T03:33:43.817Z caller=repair.go:59 component=tsdb msg="Found healthy block" mint=1645142400000 maxt=1645207200000 ulid=01FW7A67ZA3AAWGN27TBB056MF

 level=info ts=2022-02-25T03:33:43.818Z caller=repair.go:59 component=tsdb msg="Found healthy block" mint=1645207200000 maxt=1645272000000 ulid=01FW97ZS4X81MX8508Z63PA05Y

 level=info ts=2022-02-25T03:33:43.820Z caller=repair.go:59 component=tsdb msg="Found healthy block" mint=1645272000000 maxt=1645336800000 ulid=01FWB5SAW1S97SMKFWSZRWQN2Y

 level=info ts=2022-02-25T03:33:43.822Z caller=repair.go:59 component=tsdb msg="Found healthy block" mint=1645336800000 maxt=1645401600000 ulid=01FWD3JXYQHWB93E72M6GZRRCE

 level=info ts=2022-02-25T03:33:43.824Z caller=repair.go:59 component=tsdb msg="Found healthy block" mint=1645401600000 maxt=1645466400000 ulid=01FWF1CD0S511SE6T5F9H03YSD

 level=info ts=2022-02-25T03:33:43.826Z caller=repair.go:59 component=tsdb msg="Found healthy block" mint=1645466400000 maxt=1645531200000 ulid=01FWGZ608T5ER3BSYD5VC6CDNS

 level=info ts=2022-02-25T03:33:43.827Z caller=repair.go:59 component=tsdb msg="Found healthy block" mint=1645531200000 maxt=1645596000000 ulid=01FWJWZEHFZ574EX4G4J6A4R3Q

 level=info ts=2022-02-25T03:33:43.829Z caller=repair.go:59 component=tsdb msg="Found healthy block" mint=1645596000000 maxt=1645660800000 ulid=01FWMTS0K6491HCTFCZDKKEMBP

 level=info ts=2022-02-25T03:33:43.831Z caller=repair.go:59 component=tsdb msg="Found healthy block" mint=1645660800000 maxt=1645725600000 ulid=01FWPRJK5PJ91YG3RFZX9MCJC9

 level=info ts=2022-02-25T03:33:43.833Z caller=repair.go:59 component=tsdb msg="Found healthy block" mint=1645747200000 maxt=1645754400000 ulid=01FWQD4B4TY64JRJN4AQHHQV4N

 level=info ts=2022-02-25T03:33:43.835Z caller=repair.go:59 component=tsdb msg="Found healthy block" mint=1645725600000 maxt=1645747200000 ulid=01FWQD4ZW9N1MMGNZ1E5B9QR7Q

 level=info ts=2022-02-25T03:33:43.840Z caller=main.go:553 msg="Stopping scrape discovery manager..."

 level=info ts=2022-02-25T03:33:43.840Z caller=main.go:567 msg="Stopping notify discovery manager..."

 level=info ts=2022-02-25T03:33:43.840Z caller=main.go:589 msg="Stopping scrape manager..."

 level=info ts=2022-02-25T03:33:43.840Z caller=main.go:563 msg="Notify discovery manager stopped"

 level=info ts=2022-02-25T03:33:43.840Z caller=main.go:583 msg="Scrape manager stopped"

 level=info ts=2022-02-25T03:33:43.840Z caller=main.go:549 msg="Scrape discovery manager stopped"

 level=info ts=2022-02-25T03:33:43.840Z caller=manager.go:888 component="rule manager" msg="Stopping rule manager..."

 level=info ts=2022-02-25T03:33:43.840Z caller=manager.go:898 component="rule manager" msg="Rule manager stopped"

 level=info ts=2022-02-25T03:33:43.840Z caller=notifier.go:601 component=notifier msg="Stopping notification manager..."

 level=info ts=2022-02-25T03:33:43.840Z caller=main.go:755 msg="Notifier manager stopped"

 level=error ts=2022-02-25T03:33:43.841Z caller=main.go:764 err="opening storage failed: lock DB directory: resource temporarily unavailable"

因为重启prometheus时需要修改lock，但是prometheus没有权限修改data目录下的文件，所以才会报错，

# ll

-rw-r--r-- 1 nfsnobody nfsnobody     0 Jan 24 10:44 lock
-rw-r--r-- 1 nfsnobody nfsnobody 20001 Feb 25 11:33 queries.active
drwxr-xr-x 3 nfsnobody nfsnobody  4096 Feb 25 11:00 wal

解决方法就是删除lock即可，prometheus会自动创建lock，prometheus启动成功。