JanusGraph0.3.1 OLAP开发环境搭建

保证兼容性的单节点开发环境搭建

避开JanusGraph的兼容性坑，在ubuntu下使用idea进行janusgraph的OLAP开发（windows下开发会遇到兼容性问题）。使用h 做存储后端/使用spark做计算引擎。

安装环境

各安装包版本

1. centos7
2. java 1.8.0
3. scala 2.11.8 https://www.scala-lang.org/download/2.11.8.html
4. janusgraph 0.3.1 https://github.com/JanusGraph/janusgraph/releases
5. spark 2.2.0 http://archive.apache.org/dist/spark/spark-2.2.0/
6. h 1.2.6.1 http://archive.apache.org/dist/h /1.2.6.1/

主机环境

关闭防火墙

systemctl stop firewalld.service #停止firewall
systemctl disable firewalld.service #禁止firewall开机启动

安装java

下载java的rpm

rpm jdk-8u191-linux-x64.rpm

安装scala

下载scala的rpm

rpm scala-2.11.8.rpm

修改hostname

参考

 hostnamectl set-hostname centos_mini

安装单节点的standalone模式的spark

tar -zxf spark-2.2.0-bin-hadoop2.7.tgz -C /opt
cd /opt/spark-2.2.0-bin-hadoop2.7
./sbin/start-master.sh --host 10.0.10.192
./sbin/start-slave.sh spark://10.0.10.192:7077
#查看是否有进程
jps
#使用spark shell
./bin/spark-shell --master spark://10.0.10.192:7077

安装单节点h

tar -zxf h -1.2.6.1-bin.tar.gz -C /opt
cd /opt/h -1.2.6.1
vim conf/h -site. 

添加配置

<configuration>
    <property>
        <name>h .rootdir</name>
        <value>file:/opt/h 1.2.6Data</value>
    </property>
</configuration>

vim conf/h -env.sh 

添加

export JAVA_HOME=/usr/java/latest/

注释掉

export H _MASTER_OPTS=\"$H _MASTER_OPTS -XX:PermSize=128m -XX:MaxPermSize=128m\"
export H _REGIONSERVER_OPTS=\"$H _REGIONSERVER_OPTS -XX:PermSize=128m -XX:MaxPermSize=128m\"

启动

./bin/start-h .sh

测试

./bin/h  shell

解压JanusGraph到/opt目录

unzip janusgraph-0.3.1-hadoop2.zip -d /opt

idea开发环境

maven的pom. 中添加

<dependencies>
    <dependency>
        <groupId>org.janusgraph</groupId>
        <artifactId>janusgraph-all</artifactId>
        <version>0.3.1</version>
    </dependency>
</dependencies>

改配置文件

以/opt/janusgraph-0.3.1-hadoop2/conf/hadoop-graph/read-h .properties为蓝本进行修改，另存为new-read-h .properties

#
# Hadoop Graph Configuration
#
gremlin.graph=org.apache.tinkerpop.gremlin.hadoop.structure.HadoopGraph
gremlin.hadoop.graphReader=org.janusgraph.hadoop.formats.h .H InputFormat
gremlin.hadoop.graphWriter=org.apache.tinkerpop.gremlin.hadoop.structure.io.gryo.GryoOutputFormat

gremlin.hadoop.jarsInDistributedCache=true
gremlin.hadoop.inputLocation=none
gremlin.hadoop.outputLocation=output

#
# JanusGraph H  InputFormat configuration
#
janusgraphmr.ioformat.conf.storage.backend=h 
janusgraphmr.ioformat.conf.storage.hostname=10.0.10.192

#
# SparkGraphComputer Configuration
#
spark.master=spark://10.0.10.192:7077
spark.executor.memory=512m
spark.serializer=org.apache.spark.serializer.KryoSerializer
spark.kryo.registrator=org.apache.tinkerpop.gremlin.spark.structure.io.gryo.GryoRegistrator
spark.driver.memory=1g
# 为了executor能找到janus相关包
spark.executor.extraClassPath=/opt/janusgraph-0.3.1-hadoop2/lib/*

编写测试代码

    public static void main(String[] args) throws Exception {

        Graph graph = GraphFactory.open(\"new-read-h .properties\");
        GraphTraversalSource g = graph.traversal().withComputer(SparkGraphComputer.class);

        //往图中写入数据
        //generateModern(graph);
        //g.tx().commit();

        List<Vertex> it = g.V().fold().next();
        for (Vertex i : it) {
            System.out.println(i);
        }

          r = g.V().count().next();
        System.out.println(r);

        graph.close();
    }
    
    public static void generateModern(final JanusGraph graph) {
        final Vertex marko = graph.addVertex( T.label, \"person\", \"name\", \"marko\", \"age\", 29);
        final Vertex vadas = graph.addVertex(T.label, \"person\", \"name\", \"vadas\", \"age\", 27);
        final Vertex lop = graph.addVertex(T.label, \"software\", \"name\", \"lop\", \"lang\", \"java\");
        final Vertex josh = graph.addVertex(T.label, \"person\", \"name\", \"josh\", \"age\", 32);
        final Vertex ripple = graph.addVertex(T.label, \"software\", \"name\", \"ripple\", \"lang\", \"java\");
        final Vertex peter = graph.addVertex( T.label, \"person\", \"name\", \"peter\", \"age\", 35);
        marko.addEdge(\"knows\", vadas, \"weight\", 0.5d);
        marko.addEdge(\"knows\", josh, \"weight\", 1.0d);
        marko.addEdge(\"created\", lop, \"weight\", 0.4d);
        josh.addEdge(\"created\", ripple,  \"weight\", 1.0d);
        josh.addEdge(\"created\", lop, \"weight\", 0.4d);
        peter.addEdge(\"created\", lop, \"weight\", 0.2d);
    }