Welcome to the Linux Foundation Forum!

Lab3 - panic: runtime error: invalid memory address or nil pointer dereference

Options

Hi,
The network works fine in Lab 2, with both peers for Org1 running fine. However, when I add couchDb container for the peer0 I get the following error when trying to start the network.

With this command in the ./bootstrap.sh:
docker-compose -f docker-compose.yml up -d ca.example.com orderer.example.com couchdbOrg1Peer0 peer0.org1.example.com

I post two errors below, the first is from the CLI, the second from the peer0 container logs. I believe the error in CLI derives due to the error in the peer0 container, which automatically exists after trying to start.

Anyone know why the container for peer0 has an runtime error, after adding couchDb to the yaml file? The yaml file is copy-paste from course folder, so should not be any errors related to code.

The following error occur in the CLI when starting the network:

Create the channel

docker exec -e "CORE_PEER_LOCALMSPID=Org1MSP" -e "CORE_PEER_MSPCONFIGPATH=/etc/hyperledger/msp/users/Admin@org1.example.com/msp" peer0.org1.example.com peer channel create -o orderer.example.com:7050 -c allarewelcome -f /etc/hyperledger/configtx/allarewelcome.tx
2019-11-22 14:39:38.531 UTC [channelCmd] InitCmdFactory -> INFO 001 Endorser and orderer connections initialized
2019-11-22 14:39:38.551 UTC [cli.common] readBlock -> INFO 002 Received block: 0

Join peer0.org1.example.com to the channel.

docker exec -e "CORE_PEER_LOCALMSPID=Org1MSP" -e "CORE_PEER_MSPCONFIGPATH=/etc/hyperledger/msp/users/Admin@org1.example.com/msp" peer0.org1.example.com peer channel join -b allarewelcome.block
2019-11-22 14:39:38.824 UTC [channelCmd] InitCmdFactory -> INFO 001 Endorser and orderer connections initialized
Error: proposal failed (err: rpc error: code = Unavailable desc = transport is closing)

The following error is from the peer0 container logs:

2019-11-22 14:39:20.308 UTC [nodeCmd] registerDiscoveryService -> INFO 023 Discovery service activated
2019-11-22 14:39:20.309 UTC [nodeCmd] serve -> INFO 024 Starting peer with ID=[name:"peer0.org1.example.com" ], network ID=[dev], address=[peer0.org1.example.com:7051]
2019-11-22 14:39:20.309 UTC [nodeCmd] serve -> INFO 025 Started peer with ID=[name:"peer0.org1.example.com" ], network ID=[dev], address=[peer0.org1.example.com:7051]
2019-11-22 14:39:38.828 UTC [endorser] callChaincode -> INFO 026 [][e229996d] Entry chaincode: name:"cscc"
2019-11-22 14:39:38.829 UTC [ledgermgmt] CreateLedger -> INFO 027 Creating ledger [allarewelcome] with genesis block
2019-11-22 14:39:38.831 UTC [fsblkstorage] newBlockfileMgr -> INFO 028 Getting block information from block storage
panic: runtime error: invalid memory address or nil pointer dereference
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0xcecbb6]

goroutine 160 [running]:
github.com/hyperledger/fabric/core/chaincode/shim.(Handler).triggerNextState(0xc000241da0, 0x0, 0xc00009f560)
/opt/gopath/src/github.com/hyperledger/fabric/core/chaincode/shim/handler.go:35 +0x26
github.com/hyperledger/fabric/core/chaincode/shim.(
Handler).handleTransaction.func1.1(0xc000241da0, 0xc00036deb0, 0xc00009f560)
/opt/gopath/src/github.com/hyperledger/fabric/core/chaincode/shim/handler.go:247 +0x42
panic(0x106e340, 0x1e32280)
/opt/go/src/runtime/panic.go:513 +0x1b9
github.com/hyperledger/fabric/core/ledger/kvledger.(Provider).openInternal(0xc00014c780, 0xc000459d80, 0xd, 0x0, 0x0, 0x0, 0x27)
/opt/gopath/src/github.com/hyperledger/fabric/core/ledger/kvledger/kv_ledger_provider.go:157 +0xf5
github.com/hyperledger/fabric/core/ledger/kvledger.(
Provider).Create(0xc00014c780, 0xc000083700, 0x27, 0xc00036d7a8, 0x1, 0x1)
/opt/gopath/src/github.com/hyperledger/fabric/core/ledger/kvledger/kv_ledger_provider.go:117 +0x102
github.com/hyperledger/fabric/core/ledger/ledgermgmt.CreateLedger(0xc000083700, 0x0, 0x0, 0x0, 0x0)
/opt/gopath/src/github.com/hyperledger/fabric/core/ledger/ledgermgmt/ledger_mgmt.go:97 +0x17b
github.com/hyperledger/fabric/core/peer.CreateChainFromBlock(0xc000083700, 0x1378f60, 0xc00013e890, 0x1383c80, 0xc0003412c0, 0x1, 0x1)
/opt/gopath/src/github.com/hyperledger/fabric/core/peer/peer.go:436 +0x7b
github.com/hyperledger/fabric/core/scc/cscc.joinChain(0xc0004598e0, 0xd, 0xc000083700, 0x1378f60, 0xc00013e890, 0x1383c80, 0xc0003412c0, 0x0, 0x0, 0x0, ...)
/opt/gopath/src/github.com/hyperledger/fabric/core/scc/cscc/configure.go:236 +0x79
github.com/hyperledger/fabric/core/scc/cscc.(PeerConfiger).InvokeNoShim(0xc0003417c0, 0xc0004232c0, 0x2, 0x2, 0xc0000be0a0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/opt/gopath/src/github.com/hyperledger/fabric/core/scc/cscc/configure.go:164 +0x692
github.com/hyperledger/fabric/core/scc/cscc.(
PeerConfiger).Invoke(0xc0003417c0, 0x1390b60, 0xc0001fc210, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/opt/gopath/src/github.com/hyperledger/fabric/core/scc/cscc/configure.go:120 +0x29b
github.com/hyperledger/fabric/core/chaincode/shim.(Handler).handleTransaction.func1(0xc000241da0, 0xc00009f560, 0xc0002cc300)
/opt/gopath/src/github.com/hyperledger/fabric/core/chaincode/shim/handler.go:273 +0x4eb
created by github.com/hyperledger/fabric/core/chaincode/shim.(
Handler).handleTransaction
/opt/gopath/src/github.com/hyperledger/fabric/core/chaincode/shim/handler.go:242 +0x53

Comments

  • Niklaskkkk
    Options

    However, after the network has started and peer0 container exited, I can start the container manually with docker-compose command. But, when I then go into the peer0 container and run: channel list

    I receive the same error in CLI:

    nik@nik-ThinkPad-T470s:~/Desktop/fabric-samples/startFiles$ docker exec -it peer0.org1.example.com bash
    root@25ac5fb30d1f:/opt/gopath/src/github.com/hyperledger/fabric# ls
    allarewelcome.block
    root@25ac5fb30d1f:/opt/gopath/src/github.com/hyperledger/fabric# peer channel list
    2019-11-22 15:29:13.499 UTC [channelCmd] InitCmdFactory -> INFO 001 Endorser and orderer connections initialized
    Error: Failed sending proposal, got rpc error: code = Unavailable desc = transport is closing

  • Niklaskkkk
    Options

    Anyone that could know what the issue here is?

  • Niklaskkkk
    Options

    This is the log from inside the peer 0 container:

    root@25ac5fb30d1f:/opt/gopath/src/github.com/hyperledger/fabric# export CORE_PEER_MSPCONFIGPATH=/etc/hyperledger/msp/users/Admin@org1.example.com/msp
    root@25ac5fb30d1f:/opt/gopath/src/github.com/hyperledger/fabric# peer channel fetch oldest allarewelcome.block -c allarewelcome --orderer orderer.example.com:7050
    2019-11-26 09:04:06.486 UTC [channelCmd] InitCmdFactory -> INFO 001 Endorser and orderer connections initialized
    2019-11-26 09:04:06.488 UTC [cli.common] readBlock -> INFO 002 Received block: 0
    root@25ac5fb30d1f:/opt/gopath/src/github.com/hyperledger/fabric# peer channel list
    2019-11-26 09:04:17.446 UTC [channelCmd] InitCmdFactory -> INFO 001 Endorser and orderer connections initialized
    Error: Failed sending proposal, got rpc error: code = Unavailable desc = transport is closing
    root@25ac5fb30d1f:/opt/gopath/src/github.com/hyperledger/fabric# peer channel join -b allarewelcome.block
    2019-11-26 09:04:52.416 UTC [channelCmd] InitCmdFactory -> INFO 001 Endorser and orderer connections initialized
    Error: proposal failed (err: rpc error: code = Unavailable desc = transport is closing)

  • Niklaskkkk
    Options

    After error testing for several hours I believe this has to be the specific error, this is from peer0 container which exists right after the network starts. Orderer node log say nothing about any errors. this is the peer0 log:

    2019-11-26 10:29:00.236 UTC [ledgermgmt] CreateLedger -> INFO 021 Creating ledger [allarewelcome] with genesis block
    2019-11-26 10:29:00.238 UTC [fsblkstorage] newBlockfileMgr -> INFO 022 Getting block information from block storage
    panic: runtime error: invalid memory address or nil pointer dereference
    panic: runtime error: invalid memory address or nil pointer dereference
    [signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0xcecbb6]

    goroutine 156 [running]:
    github.com/hyperledger/fabric/core/chaincode/shim.(Handler).triggerNextState(0xc001dd2180, 0x0, 0xc0001b5440)
    /opt/gopath/src/github.com/hyperledger/fabric/core/chaincode/shim/handler.go:35 +0x26
    github.com/hyperledger/fabric/core/chaincode/shim.(
    Handler).handleTransaction.func1.1(0xc001dd2180, 0xc000343eb0, 0xc0001b5440)
    /opt/gopath/src/github.com/hyperledger/fabric/core/chaincode/shim/handler.go:247 +0x42
    panic(0x106e340, 0x1e32280)
    /opt/go/src/runtime/panic.go:513 +0x1b9
    github.com/hyperledger/fabric/core/ledger/kvledger.(Provider).openInternal(0xc000442880, 0xc001e8aff0, 0xd, 0x0, 0x0, 0x0, 0x27)
    /opt/gopath/src/github.com/hyperledger/fabric/core/ledger/kvledger/kv_ledger_provider.go:157 +0xf5
    github.com/hyperledger/fabric/core/ledger/kvledger.(
    Provider).Create(0xc000442880, 0xc001da8a40, 0x27, 0xc0003437a8, 0x1, 0x1)
    /opt/gopath/src/github.com/hyperledger/fabric/core/ledger/kvledger/kv_ledger_provider.go:117 +0x102
    github.com/hyperledger/fabric/core/ledger/ledgermgmt.CreateLedger(0xc001da8a40, 0x0, 0x0, 0x0, 0x0)

  • Niklaskkkk
    Options

    This is the error I receive in the couchdb container logs:

    [info] 2019-11-26T10:28:38.398968Z nonode@nohost <0.195.0> -------- Apache CouchDB has started on http://any:5986/
    [info] 2019-11-26T10:28:38.399538Z nonode@nohost <0.7.0> -------- Application couch started on node nonode@nohost
    [info] 2019-11-26T10:28:38.399838Z nonode@nohost <0.7.0> -------- Application ets_lru started on node nonode@nohost
    [info] 2019-11-26T10:28:38.428548Z nonode@nohost <0.7.0> -------- Application rexi started on node nonode@nohost
    [info] 2019-11-26T10:28:38.441617Z nonode@nohost <0.201.0> -------- open_result error {not_found,no_db_file} for _nodes
    [warning] 2019-11-26T10:28:38.441655Z nonode@nohost <0.271.0> -------- creating missing database: _nodes
    [info] 2019-11-26T10:28:38.470566Z nonode@nohost <0.201.0> -------- open_result error {not_found,no_db_file} for _dbs
    [warning] 2019-11-26T10:28:38.470600Z nonode@nohost <0.285.0> -------- creating missing database: _dbs
    [warning] 2019-11-26T10:28:38.470639Z nonode@nohost <0.284.0> -------- creating missing database: _dbs
    [info] 2019-11-26T10:28:38.478597Z nonode@nohost <0.7.0> -------- Application mem3 started on node nonode@nohost
    [info] 2019-11-26T10:28:38.478957Z nonode@nohost <0.7.0> -------- Application fabric started on node nonode@nohost
    [info] 2019-11-26T10:28:38.529437Z nonode@nohost <0.7.0> -------- Application chttpd started on node nonode@nohost
    [notice] 2019-11-26T10:28:38.553132Z nonode@nohost <0.322.0> -------- chttpd_auth_cache changes listener died database_does_not_exist at mem3_shards:load_shards_from_db/6(line:395) <= mem3_shards:load_shards_from_disk/1(line:370) <= mem3_shards:load_shards_from_disk/2(line:399) <= mem3_shards:for_docid/3(line:86) <= fabric_doc_open:go/3(line:38) <= chttpd_auth_cache:ensure_auth_ddoc_exists/2(line:187) <= chttpd_auth_cache:listen_for_changes/1(line:134)
    [error] 2019-11-26T10:28:38.555361Z nonode@nohost emulator -------- Error in process <0.323.0> with exit value:
    {database_does_not_exist,[{mem3_shards,load_shards_from_db,"_users",[{file,"src/mem3_shards.erl"},{line,395}]},{mem3_shards,load_shards_from_disk,1,[{file,"src/mem3_shards.erl"},{line,370}]},{mem3_shards,load_shards_from_disk,2,[{file,"src/mem3_shards.erl"},{line,399}]},{mem3_shards,for_docid,3,[{file,"src/mem3_shards.erl"},{line,86}]},{fabric_doc_open,go,3,[{file,"src/fabric_doc_open.erl"},{line,38}]},{chttpd_auth_cache,ensure_auth_ddoc_exists,2,[{file,"src/chttpd_auth_cache.erl"},{line,187}]},{chttpd_auth_cache,listen_for_changes,1,[{file,"src/chttpd_auth_cache.erl"},{line,134}]}]}

  • btaadmin
    Options

    What is the CORE_PEER_ADDRESS environment variable set to? If incorrect, or not set, this can lead to the error you're seeing...

  • Niklaskkkk
    Options

    @btaadmin said:
    What is the CORE_PEER_ADDRESS environment variable set to? If incorrect, or not set, this can lead to the error you're seeing...

    Hi @btaadmin ,

    I checked both compose-yaml and the env variables in container itself, seems all fine. Also the couchDB is running as expected. Here is additional information, as I am very unsure about what the issue migth be.

    core.yml:

    ledger:

    blockchain:

    state:
    # stateDatabase - options are "goleveldb", "CouchDB"
    # goleveldb - default state database stored in goleveldb.
    # CouchDB - store state database in CouchDB
    stateDatabase: CouchDB
    # Limit on the number of records to return per query
    totalQueryLimit: 100000
    couchDBConfig:
    # It is recommended to run CouchDB on the same server as the peer, and
    # not map the CouchDB container port to a server port in docker-compose.
    # Otherwise proper security must be provided on the connection between
    # CouchDB client (on the peer) and server.
    couchDBAddress: 127.0.0.1:5984
    # This username must have read and write authority on CouchDB
    username:
    # The password is recommended to pass as an environment variable
    # during start up (eg LEDGER_COUCHDBCONFIG_PASSWORD).
    # If it is stored here, the file must be access control protected
    # to prevent unintended users from discovering the password.
    password:
    # Number of retries for CouchDB errors
    maxRetries: 3
    # Number of retries for CouchDB errors during peer startup
    maxRetriesOnStartup: 12
    # CouchDB request timeout (unit: duration, e.g. 20s)
    requestTimeout: 35s
    # Limit on the number of records per each CouchDB query
    # Note that chaincode queries are only bound by totalQueryLimit.
    # Internally the chaincode may execute multiple CouchDB queries,
    # each of size internalQueryLimit.
    internalQueryLimit: 1000
    # Limit on the number of records per CouchDB bulk update batch
    maxBatchUpdateSize: 1000
    # Warm indexes after every N blocks.
    # This option warms any indexes that have been
    # deployed to CouchDB after every N blocks.
    # A value of 1 will warm indexes after every block commit,
    # to ensure fast selector queries.
    # Increasing the value may improve write efficiency of peer and CouchDB,
    # but may degrade query response time.
    warmIndexesAfterNBlocks: 1
    # Create the _global_changes system database
    # This is optional. Creating the global changes database will require
    # additional system resources to track changes and maintain the database
    createGlobalChangesDB: false

    docker-compose.yml:

    peer0.org1.example.com:
    container_name: peer0.org1.example.com
    image: hyperledger/fabric-peer
    environment:
    - CORE_VM_ENDPOINT=unix:///host/var/run/docker.sock
    - CORE_PEER_ID=peer0.org1.example.com
    - FABRIC_LOGGING_SPEC=info
    - CORE_CHAINCODE_LOGGING_LEVEL=info
    - CORE_PEER_LOCALMSPID=Org1MSP
    - CORE_PEER_MSPCONFIGPATH=/etc/hyperledger/msp/peer/
    - CORE_PEER_ADDRESS=peer0.org1.example.com:7051
    - CORE_VM_DOCKER_HOSTCONFIG_NETWORKMODE=startFiles_basic
    - CORE_LEDGER_STATE_STATEDATABASE=CouchDB
    - CORE_LEDGER_STATE_COUCHDBCONFIG_COUCHDBADDRESS=couchdbOrg1Peer0:5984
    - CORE_LEDGER_STATE_COUCHDBCONFIG_USERNAME=peer0.Org1
    - CORE_LEDGER_STATE_COUCHDBCONFIG_PASSWORD=password
    working_dir: /opt/gopath/src/github.com/hyperledger/fabric
    command: peer node start
    # command: peer node start --peer-chaincodedev=true
    ports:
    - 7051:7051
    - 7053:7053
    volumes:
    - /var/run/:/host/var/run/
    - ./crypto-config/peerOrganizations/org1.example.com/peers/peer0.org1.example.com/msp:/etc/hyperledger/msp/peer
    - ./crypto-config/peerOrganizations/org1.example.com/users:/etc/hyperledger/msp/users
    - ./config:/etc/hyperledger/configtx
    depends_on:
    - orderer.example.com
    - couchdbOrg1Peer0
    networks:
    - basic

    couchdbOrg1Peer0:
    container_name: couchdbOrg1Peer0
    image: hyperledger/fabric-couchdb
    environment:
    - COUCHDB_USER=​peer0.Org1
    - COUCHDB_PASSWORD=​password
    ports:
    - "5984:5984"
    networks:
    - basic

    From the CLI logs

    user:~/Desktop/fabric-samples$ curl http://127.0.0.1:5984/
    {"couchdb":"Welcome","version":"2.2.0","git_sha":"2a16ec4","features":["pluggable-storage-engines","scheduler"],"vendor":{"name":"The Apache Software Foundation"}}

    user:docker exec -it peer0.org1.example.com bashroot@ba0c7f2796b6:/opt/gopath/src/github.com/hyperledger/fabric# printenv
    CORE_LEDGER_STATE_COUCHDBCONFIG_COUCHDBADDRESS=couchdbOrg1Peer0:5984
    CORE_PEER_LOCALMSPID=Org1MSP
    CORE_VM_ENDPOINT=unix:///host/var/run/docker.sock
    CORE_CHAINCODE_LOGGING_LEVEL=info
    CORE_LEDGER_STATE_COUCHDBCONFIG_PASSWORD=password
    FABRIC_LOGGING_SPEC=info
    PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
    PWD=/opt/gopath/src/github.com/hyperledger/fabric
    CORE_PEER_MSPCONFIGPATH=/etc/hyperledger/msp/peer/
    CORE_VM_DOCKER_HOSTCONFIG_NETWORKMODE=startFiles_basic
    CORE_PEER_ID=peer0.org1.example.com
    SHLVL=1
    HOME=/root
    FABRIC_CFG_PATH=/etc/hyperledger/fabric
    CORE_LEDGER_STATE_COUCHDBCONFIG_USERNAME=peer0.Org1
    CORE_PEER_ADDRESS=peer0.org1.example.com:7051
    CORE_LEDGER_STATE_STATEDATABASE=CouchDB
    _=/usr/bin/printenv

  • thatgeekinit
    Options

    Is anyone actually getting their questions answered in this forum, I've been stuck for several days and my posts never get approved. I want a refund!

  • Niklaskkkk
    Options

    Hi, I found the solution:
    CouchDB has done an upgrade that is not compatible with the old fabric docker images. Please delete old peer image and pull down the new one to version 1.4.4 - latest, then re-create the network. It works now as expected.

  • Brintha18
    Options

    Hi, I've the same issue. I've pulled the latest couchdb image but still the error persists. Can someone help?

  • Niklaskkkk
    Niklaskkkk Posts: 113
    Options

    @Brintha18 said:
    Hi, I've the same issue. I've pulled the latest couchdb image but still the error persists. Can someone help?

    Hi @Brintha18 , what version is currently latest? Since Fabric has upgraded to version 2, it might be that the latest couchdb image is adapted to that. This course uses Fabric version 1.4, so I would recommend to try the couchdb version 1.4.4. Let me know how it goes.

Categories

Upcoming Training