Lab3 - panic: runtime error: invalid memory address or nil pointer dereference

Niklaskkkk

Hi,
The network works fine in Lab 2, with both peers for Org1 running fine. However, when I add couchDb container for the peer0 I get the following error when trying to start the network.

With this command in the ./bootstrap.sh:
docker-compose -f docker-compose.yml up -d ca.example.com orderer.example.com couchdbOrg1Peer0 peer0.org1.example.com

I post two errors below, the first is from the CLI, the second from the peer0 container logs. I believe the error in CLI derives due to the error in the peer0 container, which automatically exists after trying to start.

Anyone know why the container for peer0 has an runtime error, after adding couchDb to the yaml file? The yaml file is copy-paste from course folder, so should not be any errors related to code.

The following error occur in the CLI when starting the network:

Create the channel

docker exec -e "CORE_PEER_LOCALMSPID=Org1MSP" -e "CORE_PEER_MSPCONFIGPATH=/etc/hyperledger/msp/users/[email protected]/msp" peer0.org1.example.com peer channel create -o orderer.example.com:7050 -c allarewelcome -f /etc/hyperledger/configtx/allarewelcome.tx
2019-11-22 14:39:38.531 UTC [channelCmd] InitCmdFactory -> INFO 001 Endorser and orderer connections initialized
2019-11-22 14:39:38.551 UTC [cli.common] readBlock -> INFO 002 Received block: 0

Join peer0.org1.example.com to the channel.

docker exec -e "CORE_PEER_LOCALMSPID=Org1MSP" -e "CORE_PEER_MSPCONFIGPATH=/etc/hyperledger/msp/users/[email protected]/msp" peer0.org1.example.com peer channel join -b allarewelcome.block
2019-11-22 14:39:38.824 UTC [channelCmd] InitCmdFactory -> INFO 001 Endorser and orderer connections initialized
Error: proposal failed (err: rpc error: code = Unavailable desc = transport is closing)

The following error is from the peer0 container logs:

2019-11-22 14:39:20.308 UTC [nodeCmd] registerDiscoveryService -> INFO 023 Discovery service activated
2019-11-22 14:39:20.309 UTC [nodeCmd] serve -> INFO 024 Starting peer with ID=[name:"peer0.org1.example.com" ], network ID=[dev], address=[peer0.org1.example.com:7051]
2019-11-22 14:39:20.309 UTC [nodeCmd] serve -> INFO 025 Started peer with ID=[name:"peer0.org1.example.com" ], network ID=[dev], address=[peer0.org1.example.com:7051]
2019-11-22 14:39:38.828 UTC [endorser] callChaincode -> INFO 026 [][e229996d] Entry chaincode: name:"cscc"
2019-11-22 14:39:38.829 UTC [ledgermgmt] CreateLedger -> INFO 027 Creating ledger [allarewelcome] with genesis block
2019-11-22 14:39:38.831 UTC [fsblkstorage] newBlockfileMgr -> INFO 028 Getting block information from block storage
panic: runtime error: invalid memory address or nil pointer dereference
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0xcecbb6]

goroutine 160 [running]:
github.com/hyperledger/fabric/core/chaincode/shim.(Handler).triggerNextState(0xc000241da0, 0x0, 0xc00009f560)
/opt/gopath/src/github.com/hyperledger/fabric/core/chaincode/shim/handler.go:35 +0x26
github.com/hyperledger/fabric/core/chaincode/shim.(Handler).handleTransaction.func1.1(0xc000241da0, 0xc00036deb0, 0xc00009f560)
/opt/gopath/src/github.com/hyperledger/fabric/core/chaincode/shim/handler.go:247 +0x42
panic(0x106e340, 0x1e32280)
/opt/go/src/runtime/panic.go:513 +0x1b9
github.com/hyperledger/fabric/core/ledger/kvledger.(Provider).openInternal(0xc00014c780, 0xc000459d80, 0xd, 0x0, 0x0, 0x0, 0x27)
/opt/gopath/src/github.com/hyperledger/fabric/core/ledger/kvledger/kv_ledger_provider.go:157 +0xf5
github.com/hyperledger/fabric/core/ledger/kvledger.(Provider).Create(0xc00014c780, 0xc000083700, 0x27, 0xc00036d7a8, 0x1, 0x1)
/opt/gopath/src/github.com/hyperledger/fabric/core/ledger/kvledger/kv_ledger_provider.go:117 +0x102
github.com/hyperledger/fabric/core/ledger/ledgermgmt.CreateLedger(0xc000083700, 0x0, 0x0, 0x0, 0x0)
/opt/gopath/src/github.com/hyperledger/fabric/core/ledger/ledgermgmt/ledger_mgmt.go:97 +0x17b
github.com/hyperledger/fabric/core/peer.CreateChainFromBlock(0xc000083700, 0x1378f60, 0xc00013e890, 0x1383c80, 0xc0003412c0, 0x1, 0x1)
/opt/gopath/src/github.com/hyperledger/fabric/core/peer/peer.go:436 +0x7b
github.com/hyperledger/fabric/core/scc/cscc.joinChain(0xc0004598e0, 0xd, 0xc000083700, 0x1378f60, 0xc00013e890, 0x1383c80, 0xc0003412c0, 0x0, 0x0, 0x0, ...)
/opt/gopath/src/github.com/hyperledger/fabric/core/scc/cscc/configure.go:236 +0x79
github.com/hyperledger/fabric/core/scc/cscc.(PeerConfiger).InvokeNoShim(0xc0003417c0, 0xc0004232c0, 0x2, 0x2, 0xc0000be0a0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/opt/gopath/src/github.com/hyperledger/fabric/core/scc/cscc/configure.go:164 +0x692
github.com/hyperledger/fabric/core/scc/cscc.(PeerConfiger).Invoke(0xc0003417c0, 0x1390b60, 0xc0001fc210, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/opt/gopath/src/github.com/hyperledger/fabric/core/scc/cscc/configure.go:120 +0x29b
github.com/hyperledger/fabric/core/chaincode/shim.(Handler).handleTransaction.func1(0xc000241da0, 0xc00009f560, 0xc0002cc300)
/opt/gopath/src/github.com/hyperledger/fabric/core/chaincode/shim/handler.go:273 +0x4eb
created by github.com/hyperledger/fabric/core/chaincode/shim.(Handler).handleTransaction
/opt/gopath/src/github.com/hyperledger/fabric/core/chaincode/shim/handler.go:242 +0x53

Niklaskkkk

However, after the network has started and peer0 container exited, I can start the container manually with docker-compose command. But, when I then go into the peer0 container and run: channel list

I receive the same error in CLI:

nik@nik-ThinkPad-T470s:~/Desktop/fabric-samples/startFiles$ docker exec -it peer0.org1.example.com bash
root@25ac5fb30d1f:/opt/gopath/src/github.com/hyperledger/fabric# ls
allarewelcome.block
root@25ac5fb30d1f:/opt/gopath/src/github.com/hyperledger/fabric# peer channel list
2019-11-22 15:29:13.499 UTC [channelCmd] InitCmdFactory -> INFO 001 Endorser and orderer connections initialized
Error: Failed sending proposal, got rpc error: code = Unavailable desc = transport is closing

Niklaskkkk

Anyone that could know what the issue here is?

Niklaskkkk

This is the log from inside the peer 0 container:

root@25ac5fb30d1f:/opt/gopath/src/github.com/hyperledger/fabric# export CORE_PEER_MSPCONFIGPATH=/etc/hyperledger/msp/users/[email protected]/msp
root@25ac5fb30d1f:/opt/gopath/src/github.com/hyperledger/fabric# peer channel fetch oldest allarewelcome.block -c allarewelcome --orderer orderer.example.com:7050
2019-11-26 09:04:06.486 UTC [channelCmd] InitCmdFactory -> INFO 001 Endorser and orderer connections initialized
2019-11-26 09:04:06.488 UTC [cli.common] readBlock -> INFO 002 Received block: 0
root@25ac5fb30d1f:/opt/gopath/src/github.com/hyperledger/fabric# peer channel list
2019-11-26 09:04:17.446 UTC [channelCmd] InitCmdFactory -> INFO 001 Endorser and orderer connections initialized
Error: Failed sending proposal, got rpc error: code = Unavailable desc = transport is closing
root@25ac5fb30d1f:/opt/gopath/src/github.com/hyperledger/fabric# peer channel join -b allarewelcome.block
2019-11-26 09:04:52.416 UTC [channelCmd] InitCmdFactory -> INFO 001 Endorser and orderer connections initialized
Error: proposal failed (err: rpc error: code = Unavailable desc = transport is closing)

Niklaskkkk

After error testing for several hours I believe this has to be the specific error, this is from peer0 container which exists right after the network starts. Orderer node log say nothing about any errors. this is the peer0 log:

2019-11-26 10:29:00.236 UTC [ledgermgmt] CreateLedger -> INFO 021 Creating ledger [allarewelcome] with genesis block
2019-11-26 10:29:00.238 UTC [fsblkstorage] newBlockfileMgr -> INFO 022 Getting block information from block storage
panic: runtime error: invalid memory address or nil pointer dereference
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0xcecbb6]

goroutine 156 [running]:
github.com/hyperledger/fabric/core/chaincode/shim.(Handler).triggerNextState(0xc001dd2180, 0x0, 0xc0001b5440)
/opt/gopath/src/github.com/hyperledger/fabric/core/chaincode/shim/handler.go:35 +0x26
github.com/hyperledger/fabric/core/chaincode/shim.(Handler).handleTransaction.func1.1(0xc001dd2180, 0xc000343eb0, 0xc0001b5440)
/opt/gopath/src/github.com/hyperledger/fabric/core/chaincode/shim/handler.go:247 +0x42
panic(0x106e340, 0x1e32280)
/opt/go/src/runtime/panic.go:513 +0x1b9
github.com/hyperledger/fabric/core/ledger/kvledger.(Provider).openInternal(0xc000442880, 0xc001e8aff0, 0xd, 0x0, 0x0, 0x0, 0x27)
/opt/gopath/src/github.com/hyperledger/fabric/core/ledger/kvledger/kv_ledger_provider.go:157 +0xf5
github.com/hyperledger/fabric/core/ledger/kvledger.(Provider).Create(0xc000442880, 0xc001da8a40, 0x27, 0xc0003437a8, 0x1, 0x1)
/opt/gopath/src/github.com/hyperledger/fabric/core/ledger/kvledger/kv_ledger_provider.go:117 +0x102
github.com/hyperledger/fabric/core/ledger/ledgermgmt.CreateLedger(0xc001da8a40, 0x0, 0x0, 0x0, 0x0)

Niklaskkkk

This is the error I receive in the couchdb container logs:

[info] 2019-11-26T10:28:38.398968Z nonode@nohost <0.195.0> -------- Apache CouchDB has started on http://any:5986/
[info] 2019-11-26T10:28:38.399538Z nonode@nohost <0.7.0> -------- Application couch started on node nonode@nohost
[info] 2019-11-26T10:28:38.399838Z nonode@nohost <0.7.0> -------- Application ets_lru started on node nonode@nohost
[info] 2019-11-26T10:28:38.428548Z nonode@nohost <0.7.0> -------- Application rexi started on node nonode@nohost
[info] 2019-11-26T10:28:38.441617Z nonode@nohost <0.201.0> -------- open_result error {not_found,no_db_file} for _nodes
[warning] 2019-11-26T10:28:38.441655Z nonode@nohost <0.271.0> -------- creating missing database: _nodes
[info] 2019-11-26T10:28:38.470566Z nonode@nohost <0.201.0> -------- open_result error {not_found,no_db_file} for _dbs
[warning] 2019-11-26T10:28:38.470600Z nonode@nohost <0.285.0> -------- creating missing database: _dbs
[warning] 2019-11-26T10:28:38.470639Z nonode@nohost <0.284.0> -------- creating missing database: _dbs
[info] 2019-11-26T10:28:38.478597Z nonode@nohost <0.7.0> -------- Application mem3 started on node nonode@nohost
[info] 2019-11-26T10:28:38.478957Z nonode@nohost <0.7.0> -------- Application fabric started on node nonode@nohost
[info] 2019-11-26T10:28:38.529437Z nonode@nohost <0.7.0> -------- Application chttpd started on node nonode@nohost
[notice] 2019-11-26T10:28:38.553132Z nonode@nohost <0.322.0> -------- chttpd_auth_cache changes listener died database_does_not_exist at mem3_shards:load_shards_from_db/6(line:395) <= mem3_shards:load_shards_from_disk/1(line:370) <= mem3_shards:load_shards_from_disk/2(line:399) <= mem3_shards:for_docid/3(line:86) <= fabric_doc_open:go/3(line:38) <= chttpd_auth_cache:ensure_auth_ddoc_exists/2(line:187) <= chttpd_auth_cache:listen_for_changes/1(line:134)
[error] 2019-11-26T10:28:38.555361Z nonode@nohost emulator -------- Error in process <0.323.0> with exit value:
{database_does_not_exist,[{mem3_shards,load_shards_from_db,"_users",[{file,"src/mem3_shards.erl"},{line,395}]},{mem3_shards,load_shards_from_disk,1,[{file,"src/mem3_shards.erl"},{line,370}]},{mem3_shards,load_shards_from_disk,2,[{file,"src/mem3_shards.erl"},{line,399}]},{mem3_shards,for_docid,3,[{file,"src/mem3_shards.erl"},{line,86}]},{fabric_doc_open,go,3,[{file,"src/fabric_doc_open.erl"},{line,38}]},{chttpd_auth_cache,ensure_auth_ddoc_exists,2,[{file,"src/chttpd_auth_cache.erl"},{line,187}]},{chttpd_auth_cache,listen_for_changes,1,[{file,"src/chttpd_auth_cache.erl"},{line,134}]}]}

btaadmin

What is the CORE_PEER_ADDRESS environment variable set to? If incorrect, or not set, this can lead to the error you're seeing...

Niklaskkkk

@btaadmin said:
What is the CORE_PEER_ADDRESS environment variable set to? If incorrect, or not set, this can lead to the error you're seeing...

Hi @btaadmin ,

I checked both compose-yaml and the env variables in container itself, seems all fine. Also the couchDB is running as expected. Here is additional information, as I am very unsure about what the issue migth be.

core.yml:

ledger:

blockchain:

state:
# stateDatabase - options are "goleveldb", "CouchDB"
# goleveldb - default state database stored in goleveldb.
# CouchDB - store state database in CouchDB
stateDatabase: CouchDB
# Limit on the number of records to return per query
totalQueryLimit: 100000
couchDBConfig:
# It is recommended to run CouchDB on the same server as the peer, and
# not map the CouchDB container port to a server port in docker-compose.
# Otherwise proper security must be provided on the connection between
# CouchDB client (on the peer) and server.
couchDBAddress: 127.0.0.1:5984
# This username must have read and write authority on CouchDB
username:
# The password is recommended to pass as an environment variable
# during start up (eg LEDGER_COUCHDBCONFIG_PASSWORD).
# If it is stored here, the file must be access control protected
# to prevent unintended users from discovering the password.
password:
# Number of retries for CouchDB errors
maxRetries: 3
# Number of retries for CouchDB errors during peer startup
maxRetriesOnStartup: 12
# CouchDB request timeout (unit: duration, e.g. 20s)
requestTimeout: 35s
# Limit on the number of records per each CouchDB query
# Note that chaincode queries are only bound by totalQueryLimit.
# Internally the chaincode may execute multiple CouchDB queries,
# each of size internalQueryLimit.
internalQueryLimit: 1000
# Limit on the number of records per CouchDB bulk update batch
maxBatchUpdateSize: 1000
# Warm indexes after every N blocks.
# This option warms any indexes that have been
# deployed to CouchDB after every N blocks.
# A value of 1 will warm indexes after every block commit,
# to ensure fast selector queries.
# Increasing the value may improve write efficiency of peer and CouchDB,
# but may degrade query response time.
warmIndexesAfterNBlocks: 1
# Create the _global_changes system database
# This is optional. Creating the global changes database will require
# additional system resources to track changes and maintain the database
createGlobalChangesDB: false

docker-compose.yml:

peer0.org1.example.com:
container_name: peer0.org1.example.com
image: hyperledger/fabric-peer
environment:
- CORE_VM_ENDPOINT=unix:///host/var/run/docker.sock
- CORE_PEER_ID=peer0.org1.example.com
- FABRIC_LOGGING_SPEC=info
- CORE_CHAINCODE_LOGGING_LEVEL=info
- CORE_PEER_LOCALMSPID=Org1MSP
- CORE_PEER_MSPCONFIGPATH=/etc/hyperledger/msp/peer/
- CORE_PEER_ADDRESS=peer0.org1.example.com:7051
- CORE_VM_DOCKER_HOSTCONFIG_NETWORKMODE=startFiles_basic
- CORE_LEDGER_STATE_STATEDATABASE=CouchDB
- CORE_LEDGER_STATE_COUCHDBCONFIG_COUCHDBADDRESS=couchdbOrg1Peer0:5984
- CORE_LEDGER_STATE_COUCHDBCONFIG_USERNAME=peer0.Org1
- CORE_LEDGER_STATE_COUCHDBCONFIG_PASSWORD=password
working_dir: /opt/gopath/src/github.com/hyperledger/fabric
command: peer node start
# command: peer node start --peer-chaincodedev=true
ports:
- 7051:7051
- 7053:7053
volumes:
- /var/run/:/host/var/run/
- ./crypto-config/peerOrganizations/org1.example.com/peers/peer0.org1.example.com/msp:/etc/hyperledger/msp/peer
- ./crypto-config/peerOrganizations/org1.example.com/users:/etc/hyperledger/msp/users
- ./config:/etc/hyperledger/configtx
depends_on:
- orderer.example.com
- couchdbOrg1Peer0
networks:
- basic

couchdbOrg1Peer0:
container_name: couchdbOrg1Peer0
image: hyperledger/fabric-couchdb
environment:
- COUCHDB_USER=​peer0.Org1
- COUCHDB_PASSWORD=​password
ports:
- "5984:5984"
networks:
- basic

From the CLI logs

user:~/Desktop/fabric-samples$ curl http://127.0.0.1:5984/
{"couchdb":"Welcome","version":"2.2.0","git_sha":"2a16ec4","features":["pluggable-storage-engines","scheduler"],"vendor":{"name":"The Apache Software Foundation"}}

user:docker exec -it peer0.org1.example.com bashroot@ba0c7f2796b6:/opt/gopath/src/github.com/hyperledger/fabric# printenv
CORE_LEDGER_STATE_COUCHDBCONFIG_COUCHDBADDRESS=couchdbOrg1Peer0:5984
CORE_PEER_LOCALMSPID=Org1MSP
CORE_VM_ENDPOINT=unix:///host/var/run/docker.sock
CORE_CHAINCODE_LOGGING_LEVEL=info
CORE_LEDGER_STATE_COUCHDBCONFIG_PASSWORD=password
FABRIC_LOGGING_SPEC=info
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
PWD=/opt/gopath/src/github.com/hyperledger/fabric
CORE_PEER_MSPCONFIGPATH=/etc/hyperledger/msp/peer/
CORE_VM_DOCKER_HOSTCONFIG_NETWORKMODE=startFiles_basic
CORE_PEER_ID=peer0.org1.example.com
SHLVL=1
HOME=/root
FABRIC_CFG_PATH=/etc/hyperledger/fabric
CORE_LEDGER_STATE_COUCHDBCONFIG_USERNAME=peer0.Org1
CORE_PEER_ADDRESS=peer0.org1.example.com:7051
CORE_LEDGER_STATE_STATEDATABASE=CouchDB
_=/usr/bin/printenv

thatgeekinit

Is anyone actually getting their questions answered in this forum, I've been stuck for several days and my posts never get approved. I want a refund!

Niklaskkkk

Hi, I found the solution:
CouchDB has done an upgrade that is not compatible with the old fabric docker images. Please delete old peer image and pull down the new one to version 1.4.4 - latest, then re-create the network. It works now as expected.

Brintha18

Hi, I've the same issue. I've pulled the latest couchdb image but still the error persists. Can someone help?

Niklaskkkk

@Brintha18 said:
Hi, I've the same issue. I've pulled the latest couchdb image but still the error persists. Can someone help?

Hi @Brintha18 , what version is currently latest? Since Fabric has upgraded to version 2, it might be that the latest couchdb image is adapted to that. This course uses Fabric version 1.4, so I would recommend to try the couchdb version 1.4.4. Let me know how it goes.