Alluxio使用——TensorFlow篇

1.安装部署

TensorFlow安装部署
Alluxio安装部署

SSH免密登陆

2.使用

1).创建alluxio根目录

[bigdata@carbondata alluxio-2.0.0]$ ./bin/alluxio fs mkdir /training-dataSuccessfully created directory /training-data

2).创建本地目录，并挂载到alluxio根目录

a).创建本地目录

mkdir -p /home/bigdata/data

b).挂载到alluxio根目录

[bigdata@carbondata alluxio-2.0.0]$ ./integration/fuse/bin/alluxio-fuse mount /home/bigdata/data /training-dataStarting alluxio-fuse on local host.Successfully mounted Alluxio path "/training-data" to /home/bigdata/data.See /home/bigdata/alluxio-2.0.0/logs/fuse.log for logging messages

c).验证挂载状态

[bigdata@carbondata alluxio-2.0.0]$ ./integration/fuse/bin/alluxio-fuse statpid     mount_point     alluxio_path11074   /home/bigdata/data      /training-data

3).准备测试数据

a).下载测试数据

inception-2015-12-05

wget http://download.tensorflow.org/models/image/imagenet/inception-2015-12-05.tgz

b).测试数据上传至alluxio

## 创建数据目录[bigdata@carbondata alluxio-2.0.0]$ ./bin/alluxio fs mkdir /training-data/imagenetSuccessfully created directory /training-data/imagenet## 上传数据[bigdata@carbondata alluxio-2.0.0]$ ./bin/alluxio fs copyFromLocal /home/bigdata/inception-2015-12-05.tgz /training-data/imagenet Failed to cache: There is no worker with enough space for a new block of size 536,870,912[bigdata@carbondata alluxio-2.0.0]$ ./bin/alluxio fs copyFromLocal /home/bigdata/inception-2015-12-05.tgz /training-data/imagenet Copied file:///home/bigdata/inception-2015-12-05.tgz to /training-data/imagenet

4).图像识别测试

下载脚本

curl -o classify_image.py -L https://raw.githubusercontent.com/tensorflow/models/master/tutorials/image/imagenet/classify_image.py

运行脚本

(tensorflow) [bigdata@carbondata tensorflow_data]$ python classify_image.py --model_dir /home/bigdata/data/imagenetWARNING:tensorflow:From classify_image.py:227: The name tf.app.run is deprecated. Please use tf.compat.v1.app.run instead.WARNING:tensorflow:From classify_image.py:139: The name tf.gfile.Exists is deprecated. Please use tf.io.gfile.exists instead.W0829 21:25:37.012851 139730996795200 deprecation_wrapper.py:119] From classify_image.py:139: The name tf.gfile.Exists is deprecated. Please use tf.io.gfile.exists instead.WARNING:tensorflow:From classify_image.py:141: __init__ (from tensorflow.python.platform.gfile) is deprecated and will be removed in a future version.Instructions for updating:Use tf.gfile.GFile.W0829 21:25:39.488382 139730996795200 deprecation.py:323] From classify_image.py:141: __init__ (from tensorflow.python.platform.gfile) is deprecated and will be removed in a future version.Instructions for updating:Use tf.gfile.GFile.WARNING:tensorflow:From classify_image.py:125: The name tf.GraphDef is deprecated. Please use tf.compat.v1.GraphDef instead.W0829 21:25:41.580853 139730996795200 deprecation_wrapper.py:119] From classify_image.py:125: The name tf.GraphDef is deprecated. Please use tf.compat.v1.GraphDef instead.2019-08-29 21:26:31.840733: W tensorflow/core/ work/op_def_util.cc:357] Op BatchNormWithGlobalNormalization is deprecated. It will cease to work in GraphDef version 9. Use tf.nn.batch_normalization().WARNING:tensorflow:From classify_image.py:146: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.W0829 21:27:04.229350 139730996795200 deprecation_wrapper.py:119] From classify_image.py:146: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.2019-08-29 21:27:04.889623: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA2019-08-29 21:27:05.865288: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 3600000000 Hz2019-08-29 21:27:05.888049: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x557e570 executing computations on platform Host. Devices:2019-08-29 21:27:05.888110: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): <undefined>, <undefined>2019-08-29 21:27:07.301608: W tensorflow/core/ work/allocator.cc:107] Allocation of 8257536 exceeds 10% of system memory.2019-08-29 21:27:07.357415: W tensorflow/core/ work/allocator.cc:107] Allocation of 8257536 exceeds 10% of system memory.2019-08-29 21:27:07.359446: W tensorflow/core/ work/allocator.cc:107] Allocation of 8257536 exceeds 10% of system memory.2019-08-29 21:27:07.431136: W tensorflow/core/ work/allocator.cc:107] Allocation of 8257536 exceeds 10% of system memory.2019-08-29 21:27:07.593467: W tensorflow/core/ work/allocator.cc:107] Allocation of 8257536 exceeds 10% of system memory.2019-08-29 21:27:08.113432: W tensorflow/compiler/jit/mark_for_compilation_pass.cc:1412](One-time warning): Not using XLA:CPU for cluster because envvar TF_XLA_FLAGS=--tf_xla_cpu_global_jit was not set.  If you want XLA:CPU, either set that envvar, or use experimental_jit_scope to enable XLA:CPU.  To confirm that XLA is active, pass --vmodule=xla_compilation_cache=1 (as a proper command-line flag, not via TF_XLA_FLAGS) or set the envvar XLA_FLAGS=--xla_hlo_profile.WARNING:tensorflow:From classify_image.py:85: The name tf.gfile.GFile is deprecated. Please use tf.io.gfile.GFile instead.W0829 21:27:11.583446 139730996795200 deprecation_wrapper.py:119] From classify_image.py:85: The name tf.gfile.GFile is deprecated. Please use tf.io.gfile.GFile instead.giant panda, panda, panda bear, coon bear, Ailuropoda melanoleuca (score = 0.89107)indri, indris, Indri indri, Indri brevicaudatus (score = 0.00779)lesser panda, red panda, panda, bear cat, cat bear, Ailurus fulgens (score = 0.00296)custard apple (score = 0.00147)earthstar (score = 0.00117)